- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a controlled comparison of MAE, JEPA, and autoregressive self-supervised pretraining against training from scratch for intracortical neural decoding. Using a shared BIT Transformer backbone, the study compares pretraining on human-only and combined human–nonhuman-primate corpora, comprising approximately 208 and 345 hours of data, respectively, and evaluates both subject-specific and joint fine-tuning. Downstream tasks include attempted and imagined speech, handwriting, and typing, evaluated using phoneme error rate (PER) or character error rate (CER) with greedy CTC decoding. The study uses a fixed hyperparameter search budget for fine-tuning and conducts five-seed stability analyses of selected configurations. The results show that the benefits of pretraining vary across objectives and datasets, adding nonhuman-primate data does not consistently improve performance, and carefully tuned models trained from scratch remain competitive under the evaluated settings.
Critical strengths
-
A fair comparison with tuned scratch baselines. The study uses a shared BIT Transformer backbone and allocates the same fine-tuning hyperparameter search budget to pretrained and randomly initialized models. This makes it easier to assess whether pretraining provides gains beyond careful task-specific training, rather than gains that may simply reflect differences in tuning effort.
-
Evaluation across tasks and pretraining data compositions. The study covers speech, handwriting, and typing, and compares different pretraining objectives and corpus compositions. The results show that pretraining does not consistently improve decoding performance, and adding nonhuman-primate recordings does not reliably benefit human decoding. These findings caution against assuming that pretraining, particularly with data from additional species, will necessarily improve neural decoding.
Critical weaknesses
-
The stability analysis does not adequately establish the robustness of the main conclusion. The analysis uses only five fine-tuning seeds per selected configuration and removes high-error clusters based on the observed outcomes (Section 3; Figure 3). Although unsuccessful runs are retained in the main hyperparameter sweep, excluding high-error seeds from the stability analysis may bias the estimated differences between pretrained and scratch models. With only five seeds, it is also difficult to characterize training variability reliably. In addition, the absence of statistically significant improvements does not establish that pretraining offers no meaningful benefit. Reporting results with all seeds retained, along with failure rates and uncertainty estimates for effect sizes, would clarify how sensitive the conclusions are to the treatment of failed runs.
-
The data separation protocol is insufficiently specified to verify the evaluation setting. Appendix A.3 describes the downstream data splits but does not clearly state whether recordings used for downstream validation and testing are excluded from self-supervised pretraining. Appendix A.2 also notes that normalization statistics are computed across all available sessions of the same subject where possible, leaving it unclear whether these statistics cross downstream split boundaries. Furthermore, some datasets come from the same biological participants, complicating the interpretation of evidence across datasets as independent. These omissions do not demonstrate data leakage, but they prevent a clear assessment of how strictly the pretraining and evaluation data are separated, weakening confidence in the reported benefits of pretraining.
Questions for authors
-
Data separation — clarification: Were downstream validation and test recordings excluded from SSL pretraining and from the computation of normalization statistics? How was overlap between datasets from the same participant handled, and what unit—trial, session, or recording day—was used to define each split?
-
Seed exclusion and training reliability — clarification and robustness analysis: How many fine-tuning seeds were excluded from each configuration, and according to what criteria? Do the conclusions hold when all five seeds from the stability analysis are retained? Please report the results with all seeds included, failure rates, and uncertainty estimates for the pretraining-versus-scratch differences. How many independent pretraining runs were performed for each objective–corpus combination? If multiple pretrained checkpoints are available, how much do downstream results vary across them? These details would help distinguish performance conditional on successful training from the reliability of the overall training procedure.
-
Statistical evaluation — clarification: Were the statistical comparisons in Figure 3 based on validation or held-out test errors? Appendix F reports validation errors. Please specify the baseline used to calculate the relative error reductions in Figure 3 and the baseline used in the statistical tests, particularly for pretrained models under joint fine-tuning. Do the conclusions hold when pretrained models are compared with their corresponding scratch baselines on held-out test data?
-
Independent participants and joint fine-tuning — clarification: Table 1 suggests that the downstream evaluation instances do not represent ten distinct biological participants, and both handwriting datasets include data from T5. Which results support the benefits of pooling data across different participants, rather than pooling datasets from the same participant? Can the current experiments distinguish these effects, and would this distinction change the interpretation of the joint fine-tuning results?
-
Effect of adding NHP data — clarification and supporting analysis: How were human and NHP recordings sampled during pretraining, and did adding NHP data change the number of optimization steps or the exposure to human recordings? Table 5 reports that MAE reconstruction on the diagnostic pool decreases from 0.093 with human-only pretraining to 0.021 with human-plus-NHP pretraining. Do the available per-dataset diagnostics help distinguish negative transfer from optimization difficulties or sampling imbalance? Does the lower reconstruction correspond to consistent changes in downstream decoding performance across tasks, or do the diagnostic and downstream results suggest different effects of adding NHP data?
7 comments
W2) We thank the reviewer for raising this point; we have clarified the protocol in Appendix A.2–A.3. Self-supervised pre-training uses only the training split of each downstream dataset. The downstream validation and test recordings are never seen during pre-training. In addition to the training splits, pre-training uses the unlabelled official competition hold-out recordings of Card et al. and Willett et al. These are separate files from those from which our validation and test sets are drawn. The pre-training checkpoint is selected on the validation split of two subjects (Willett T12 and Card), and the fine-tuning configuration on the validation split of each subject. The test set is used only once, for the final evaluation. Normalisation uses only label-free per-channel statistics. Features are z-scored within each recording session. Regarding the datasets coming from the same biological participant: this happens for the following pairs: Card and Kunz T15, Willett and Kunz T12. These two pairs of subjects are always trained jointly (see also section 2.5 of the manuscript, paragraph “Single vs Joint Fine-Tuning”). While we agree that pooling together datasets coming from different subjects performing the same task is different compared to pooling together datasets coming from the same subject performing the same task, we believe that the following experiment disentangles the confusion. The joint-task experiment pools subjects performing the same task (speech, handwriting, typing), and its results can be compared to the pseudo-joint results of the pairs Card + Kunz T15 and Willett + Kunz T12. Please see in particular the answer to Q4 below.
Answer to Reviewer's Questions
Q1) Please see our answer to Weakness 1 above.
Q2) We thank the reviewer for this question. Please see also our answer to Weakness 1. A seed is considered failed if all of the following hold: the distribution of the 10 seeds of that configuration is bimodal (Silverman test, α = 0.10); the seed belongs to the upper cluster of a 2-means split; the ratio between the upper and lower clusters’ mean exceeds 1.25. Regarding the last question of pretraining variability: for each objective, model size and corpus we ran a hyperparameter sweep with a single seed per configuration. This amounts to 60 configurations for MAE and JEPA and 12 for AR, i.e. 180 MAE, 180 JEPA and 36 AR runs per corpus over three model sizes, about 19,000 GPU-hours in total only for pretraining. One checkpoint per objective-corpus combination was selected on the pre-training validation criterion and fine-tuned. We did not repeat pre-training with different seeds, and we did not fine-tune multiple checkpoints of the same combination, unfortunately, because of the extremely high computational cost.
Q3) Thanks for allowing us to clarify. The results in Figure 3 are reported on the validation set in all cases. The baseline of Figure 3 and subsequent analysis is the stability run of the model trained from scratch. Since then, we have repeated the analysis with n=10 stability seeds and added the comparison of the joint fine-tuning models with the joint from-scratch baseline (please see answer to Weakness 1 above). We repeated the full stability analysis evaluating on the held-out test set. The conclusions are unchanged. Exactly the same 137 seeds are identified as failed on validation and test, so failure rates are identical. Across the 130 pretrained-versus-scratch and joint-versus-single comparisons, the effect has the same sign in 121. No comparison is significantly better than scratch on one split and significantly worse on the other. Pre-training is significantly better than scratch in 6 of 60 single-subject comparisons on test (8 on validation) and in 9 of 60 joint comparisons (10 on validation). JEPA remains worse than scratch throughout. MAE-HM again lowers the median error of the scratch joint model on all 10 subjects. The only comparisons that lose significance on test are small effects or subjects with very small test sets (e.g. 25 sentences for Kunz T15), where test-set noise reduces statistical power.
Q4) We thank the reviewer for the question. Two experiments disentangle these effects. First, we compare the pseudo-joint fine-tuning (Card+KunzT15, Willett+KunzT12) to the joint-speech models. Adding data of different participants performing the same task is beneficial. Second, on typing: data comes from two separate participants (T17 and T18). Also in this case, training jointly on the available typing data from both participants is beneficial.
Q5) We thank the reviewer for raising this point. The training data was shuffled across human and non-human primates. The addition of NHP data does not change the exposure to human data, as we fix the total number of pre-training epochs to 400 regardless of the training corpus.
Lower reconstruction R² after adding NHP data does not necessarily indicate a worse model. With more and more heterogeneous data, a model of fixed capacity spreads its reconstruction ability over many more sources, so its fit to each individual dataset becomes more diluted. We report the MAE reconstruction R² on human and NHP datasets (final checkpoints; previous Table 5 values come from earlier runs):
| Dataset | MAE R² (H / HM) |
|---|---|
| Human | |
| Speech (Wairagkar) | 0.36 / 0.31 |
| Handwr. (Fan) | 0.02 / -0.42 |
| Handwr. (Willett) | 0.02 / -0.75 |
| Typing (T18) | 0.07 / -0.18 |
| NHP (HM only) | |
| O'Doherty | – / -0.09 |
| Churchland | – / 0.03 |
| Even-Chen | – / 0.06 |
| Ma | – / -0.94 |
| Temmar | – / 0.01 |
The HM model's reconstruction R² is low on the NHP datasets as well, consistent with its capacity being spread across many sources rather than traded from human to NHP data. Yet after fine-tuning it decodes better than the H model on every human dataset except Kunz T15 under task-joint fine-tuning.
W1
W1) We thank the reviewer for this question. In the revised manuscript we have increased the stability analysis from 5 to 10 seeds per configuration (14 model–regime combinations × 10 subjects = 140 configurations, 1400 fine-tuning runs) and extended it to joint fine-tuning: each pre-trained checkpoint, fine-tuned jointly on all subjects of a task, is compared against joint fine-tuning from scratch (new Figure 3 in the main text and Figure 8 in Appendix G).
137 of 1400 runs (9.8%) were excluded. Failures are concentrated on typing and handwriting: 129 of 560 runs (23%), in 52 of 56 configurations. On speech only 8 of 840 runs failed (1%), in 6 of 84 configurations, 4 of which are Jude anarthria. The per-configuration counts and excluded values are reported in Appendix Table 10 of the revised manuscript and in the table below.
Failure rates per model and fine-tuning regime, pooled over the 10 subjects:
Convergence failures are therefore a property of the tasks (typing and handwriting, with small datasets), and pre-training neither causes nor prevents them.
For completeness, we report here the full analysis without excluding any seed. As requested, for each subject and model we report four quantities: the relative change of the median error with respect to scratch, with a percentile-bootstrap 95% confidence interval (CI; 10,000 resamples of seeds within each group); the probability of superiority, i.e. the probability that a seed of the pre-trained model reaches a lower error than a seed of the scratch model, with its bootstrap CI; the Mann–Whitney U test with Benjamini–Hochberg (BH) correction.
Retaining all seeds almost never changes the direction of an effect. The sign of the median change flips in only 3 of 60 single-subject comparisons, 1 of 60 joint comparisons and none of the 10 scratch single-vs-joint comparisons. Only two of these flips are substantial, both on Jude T17 with single-subject fine-tuning. There, at least half of the seeds failed, so the median falls among the failed runs. The other two flips are small effects (below 3%) close to zero. Significance decreases, because the retained failures widen the distributions. With failed seeds excluded, pre-training is significantly better than scratch in 8 and significantly worse in 38 of the 60 single-subject comparisons; with all seeds retained, these numbers become 4 and 34. In joint fine-tuning they go from 10 better and 23 worse to 7 better and 15 worse. For the comparison between scratch joint and scratch single training, they go from 5 better and 2 worse to 1 better and none worse. Most of the comparisons that lose significance (14 of 19) are on typing and handwriting, where failures are concentrated.
The bootstrap confidence intervals, computed with all seeds retained, lead to the same conclusions. In single-subject fine-tuning, the CI excludes zero in favour of pre-training in only 4 of 60 comparisons: AR-H on Kunz T12 and T18, AR-HM on T18, and MAE-HM on Willett handwriting. It excludes zero against pre-training in 30 comparisons, and includes zero in the remaining 26. JEPA accounts for 14 of the 30 unfavourable comparisons, being worse than scratch on every speech subject and on T18. The remaining 16 are all on speech subjects: 6 for MAE-H, 4 for AR-H, 4 for AR-HM and 2 for MAE-HM.
In joint fine-tuning, the CI excludes zero in favour of pre-training in 10 of 60 comparisons, all on speech subjects: AR-H on Wairagkar, Kunz T15, Willett T12 and Kunz T12; AR-HM on Wairagkar, Kunz T15 and Kunz T12; MAE-HM on Willett T12 and Kunz T12; and MAE-H on Kunz T12. It excludes zero against pre-training in 15 comparisons, all JEPA, and includes zero in the remaining 35. MAE-HM lowers the median error of the scratch joint model on all 10 subjects, both with and without seed exclusion. When scratch joint training is compared with scratch single-subject training, the CI excludes zero in favour of joint training on Card and Jude anarthria and against it on Kunz T15 and Kunz T12, and includes zero on the other six subjects.
In conclusion, retaining the failed seeds does not change our conclusion that pre-training offers no consistent benefit over training from scratch in single-subject fine-tuning, where differences are mostly within seed-to-seed variability or favour scratch, and that its advantages, when present, emerge in joint fine-tuning on speech, most consistently for MAE pre-trained on humans and monkeys.
Further details regarding the stability analysis are provided in the answer to Question 1 below.