acceptodds
Under review as a conference paper at ICLR 2027

CorticAll: A Universal Intracortical Decoder For Communication BCIs

Abstract

Intracortical brain–computer interfaces (BCIs) for communication remain difficult to scale across participants because their input is heterogeneous: individuals differ in the number and placement of implanted arrays, and each channel samples a distinct neural population. Decoders are therefore typically trained one participant at a time, forgoing the benefits of pooled data. We introduce CorticAll, a multimodal intracortical decoder that accepts heterogeneous channel sets and supports speech, handwriting, and typing decoding alongside cursor control. A shared Channel Tokeniser processes each channel, Channel Identity embeddings preserve its identity, and a cross-attention Channel Mixer maps any number of channels to a fixed-size representation, enabling end-to-end training across datasets. We evaluate CorticAll on 11 datasets from 7 participants, spanning 4 cortical regions and 4 modalities (≈173 h). We further introduce supervised-on-scale, joint supervised training across datasets and modalities, and study it for pooled decoding, fine-tuning, and transfer. Fine-tuning the pooled model reaches state-of-the-art error rates on 7 of the 8 datasets with published baselines and decodes cursor velocity at up to R2 = 0.93. Finally, latent-space analyses reveal substantial shared structure across modalities and participants, suggesting that the model reuses common functional primitives across both.

Primary area:
applications to neuroscience & cognitive science
Keywords:
brain-computer interface, multimodal, transformer, cross-attention, neural decoder, crossparticipant

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_XWoFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

I am not a BCI expert

The goal is to great a multi-modal multi-participant cortical decoder trained on handwriting, speech, cursor, and typing. It can handle variable channel counts. The novelty comes from the training across modalities as the components are all previously descirbed. It also aggregates pretty much all available Utah array data.

Critical strengths

The multimodal training is a strength and novel The large (for the available data) data set is a strength The engineering is nice to bring it all together

Critical weaknesses

The SOTA claims are week given the small number of seeds and the small number of validation trials. Boot strappingn is needed to make strong claims.

The cursor targets are the original decoded targets, not the indented. This seems circular and of unclear value.

I don't understand the claims of "universal" as each new individual would require embeddings and fine-tuning. We don't see few shot data.

Questions for authors

Please use more seeds and better estimates of confidence bounds. Increase the size of the vlidation set. Please make clear what is needed for each new participant and lessen claims of "universailty"

2 comments

We thank the reviewer for the review and for the clear list of concerns, each of which led to an addition.

  1. State-of-the-art comparison, seeds and validation sets

Our comparison follows the standard practice on these datasets: we train and evaluate on the same splits as the published studies and compare the resulting numbers one to one. On that basis CorticAll is numerically better on 7 of 8 datasets. Since the validation sets are fixed by the original releases and cannot be enlarged, we added bootstrap confidence intervals as a proxy for more validation trials: they show how much each dataset result could move with a different sample of trials (10,000 resamples of the validation trials; error in %, 95% interval):

DatasetPublishedCorticAll
Card T15 speech7.17.0 [6.6, 7.5]
Willett T12 speech14.413.7 [12.9, 14.5]
Kunz T12 speech18.113.9 [11.2, 16.9]
Kunz T15 speech17.913.6 [9.9, 17.8]
Jude T17 speech34.025.0 [21.8, 28.1]
Wairagkar T15 speech7.03.8 [2.9, 4.8]
Willett T5 handwriting5.47.1 [6.1, 8.6]
Fan T5 handwriting19.214.9 [13.8, 16.0]
Jude T17 typing27.923.0 [21.1, 25.0]
Jude T18 typing8.14.7 [3.8, 5.5]

For five of the seven datasets (counting the Kunz and the Jude typing releases once each) the advantage is robust to resampling of the validation trials; for Card and Willett speech the published value lies within our interval. Intervals for every result are in App. A.7. Additional seeds were not feasible within the rebuttal period, since each seed requires retraining every regime on every dataset (App. A.6).

  1. Cursor targets

As stated in Section 2.1, the cursor targets are the trajectories decoded in the source studies. These datasets were recorded during online cursor control: the participants steered a decoded cursor to on-screen targets, and the cursor's trajectory is the only movement signal that was recorded. The revision adds why we include cursor: to test whether a continuous control modality can share weights with speech, handwriting and typing. It can, with R² up to 0.93.

  1. "Universal" and what a new participant needs

CorticAll is universal in its inputs and outputs: it accepts any configuration of intracortical arrays and channel sets, and decodes the four major communication BCI modalities with one set of weights. This is what allows a single model to be trained jointly across participants and modalities, the purpose it was built for; the definition is now given where the term is first used. Adding a participant who was not part of training requires labelled data from them: new channel embeddings are learned and the rest of the model is fine-tuned. New experiments with participants held out of pretraining show what this takes (Table 2, App. A.10). PER (%) at full target data, by which parameters are trained:

TrainedCard T15Wairagkar T15Kunz T15Willett T12Kunz T12Kunz T16Jude T17Kunz T17
None (zero-shot)83858480858279195
Embeddings only66.666.781.879.492.180.494.460.7
Front-end (1.3M)13.512.125.726.448.031.035.632.4
All (14.1M)9.58.330.417.334.731.732.233.5
From scratch8.612.727.014.626.924.235.035.9

New channel embeddings alone are not enough; the spatial front-end has to adapt as well. Decoding rises above chance only from about 30 to 60 minutes of labelled data (Figure 4). Beyond that, pretraining on other participants reduces the data needed for some participants and not for others (Figure 4, App. A.10).

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer,

As the discussion period is approaching its deadline, we would appreciate any feedback on our rebuttal, particularly whether it addresses your concerns. We would be happy to clarify any remaining points.

Thank you, Authors

Reviewer_95KVOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

CorticAll is an interesting, and latent structure analysis work, and does seem to capture some shared latent for motor controls in typing, speaking, and hand writing. This idea is substantiated well, and the main contribution is multi task decoding for BCI.

Critical strengths

The paper motivates supervised-on-scale as a more data-efficient alternative to self-supervised pretraining and frames the pretrained representation as a path toward foundation-model-style transfer. This idea is substantiated well, the main contribution is multi task decoding.

Critical weaknesses

Extrapolating tasks and having benefits from it is not new to ML community (not even for neural decoding), so one intellectual contribution is the channel invariance, but it is not well supported, especially in the perspectives for subject adaptation/generalization.

All fine-tuning and leave-one-out experiments use the full training split of each target dataset; without a learning curve over target-data quantity (e.g., fine-tuning on 5–100% of the target split, or a matched calibration-time budget as in NDT2/NDT3), the central claims about data efficiency and practical adaptation cost to a new participant remain unsupported.

Channel embedding is tested through ablation but it breaks because high dependency to that part, not showing the entire performance requires such logic. For example, the architecture can be as simple as regular transformer as shown by Fogg et al. ("A  generalizable speech neuroprosthesis").

Also, it is unsure which part is actually updated. If it is only channel embedding, two questions remain, how it works without fine-tuning. (with random, or zero-initialized channel embedding), how it works only fine-tuning channel embedding.

Questions for authors

See weaknesses

2 comments

We thank the reviewer for the assessment and for the experiments suggested, which are now in the paper. We follow the order of the review.

  1. Contribution and support for the channel-flexible front-end

We agree that benefiting from multiple tasks is not new in itself. Multi-task decoding is one of the paper's main contributions, alongside the architecture and the latent-space analysis: to our knowledge, no prior model has been trained jointly with supervision across BCI communication modalities on human intracortical data, decoding speech, handwriting and typing as alignment-free sequences together with cursor control from one set of weights. The revision also goes beyond combining modalities and analyses how they interact (Figure 3, App. A.9).

The purpose of the channel-flexible front-end is to let the model scale across datasets without significant growth in parameters: 128 per electrode, against about 3,000 for a dataset-specific projection. In that role it gives lower error than dataset-specific input layers, as shown in the ablation below. The reviewer notes that it is not well supported with respect to adaptation to new subjects. The original submission held out datasets, not participants, so it could not answer this question. The revision replaces that experiment with leave-one-participant-out pretraining and examines how much target data is needed and which parameters must adapt; both are reported below.

  1. Learning curves over target-data quantity

We fine-tuned on 5 minutes to 8 hours of target data for the largest dataset of three participants held out of pretraining, against training from scratch on the same subsets (Figure 4, App. A.10; three seeds up to 4 hours). Minutes of target data needed to reach a given PER (from scratch / fine-tuned, interpolated from the curves):

PERCard T15Willett T12Jude T17
60%121 / 5022 / 4293 / 111
50%183 / 8651 / 70243 / 191
40%280 / 153120 / 120441 / 323

Pretraining roughly halves the data T15 needs, helps T17 at lower error levels, and does not help T12; no regime decodes from 5 to 15 minutes. One reading of this participant dependence is that the present corpus covers some, but not all, of the neural patterns a new participant brings, so transfer succeeds where they are covered. With four speech participants this remains a hypothesis.

  1. The channel-embedding ablation and Fogg et al.

We now cite and discuss Fogg et al. in the paper. The reviewer is right that removing channel identity from a model built around it shows dependence, not necessity. The informative comparison is with the design that needs no channel identity: a per-dataset input projection, as in Fogg et al. and BIT. Our ablation includes it, with the same Temporal Transformer and training pipeline (App. A.2; all-modality training, single seed, best checkpoint):

Front-endSpeech PERHandwriting CERTyping CERCursor R²Parameters
Per-dataset projection14.514.617.70.5823.0M
CorticAll (shared)12.811.311.80.6314.3M

The core of the two models is very similar: a transformer over time operating on 80 ms tokens. With per-dataset projections the input stage is trained on one dataset only. CorticAll's shared front-end is more complex, and in exchange it is trained end to end on all data. Channel identity is the mechanism that allows this sharing, since channels are not exchangeable. Fogg et al. train on speech only, and our projection baseline is comparable in kind, though not identical, to their input layers.

  1. Which parameters are updated; no fine-tuning; embeddings only

For each speech participant we pretrained a model without them and compared what must be trained on the new participant (Table 2, App. A.10). PER (%) at full target data:

TrainedCard T15Wairagkar T15Kunz T15Willett T12Kunz T12Kunz T16Jude T17Kunz T17
None (zero-shot)83858480858279195
Embeddings only66.666.781.879.492.180.494.460.7
Front-end (1.3M)13.512.125.726.448.031.035.632.4
All (14.1M)9.58.330.417.334.731.732.233.5
From scratch8.612.727.014.626.924.235.035.9

Without fine-tuning the model does not decode, with either random or mean-initialised channel embeddings. Training the embeddings alone gives weak decoding on three datasets and none elsewhere. Adapting the spatial front-end with the Temporal Transformer frozen restores decoding on every dataset, which parallels the adaptation of an input projection onto a frozen decoder in Fogg et al. Fine-tuning all parameters after an embedding warm-up is the most robust. In fine-tuning within the training pool (Table 1), all parameters are updated.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer,

As the discussion period is approaching its deadline, we would appreciate any feedback on our rebuttal, particularly whether it addresses your concerns. We would be happy to clarify any remaining points.

Thank you, Authors

Reviewer_k7BcOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes CorticAll, a unified model for various intracortical human neural decoding tasks such as speech, handwriting, typing, and cursor decoding. The proposed model can process data of heterogeneous channel counts due to channel-specific tokenisation, following which cross-attention and a temporal transformer backbone process them further before reading out to behavioural outputs. The model is trained jointly in a supervised manner on datasets from each task subset or all task subsets, and further finetuning is done on individual datasets either in a leave-dataset-out fashion or in a continual dataset-specific training manner. In general, it seems like the model outperforms prior baselines where available and does well when continually trained after pretraining on the same task or all tasks. The authors also carry out some latent space analyses to see whether shared structure exists across the datasets that the models may exploit to learn better joint representations.

Critical strengths
  • The paper is mostly clear and a useful effort in pretraining a single decoder on several human-relevant BCI tasks.
  • There are good performance gains on several datasets versus the stated SoTA approaches.
Critical weaknesses
  • On several datasets, it seems like the all-modality pretrained model doesn't outperform the within-modality model. This suggests that either the hyperparameters need to be tuned better, or there needs to be some data selection strategy, and/or that some modalities and datasets hurt performance on others. Could the authors explain this more and clarify why especially for speech tasks even in the finetuned case it seems like modality-specific models are better on several datasets? Also in some cases, unseen finetuning is worse than a single scratch model. I'm curious about those results, and also why there's no leave dataset out finetuning for non-speech datasets.
  • Some tasks do not have associated baselines, which makes it hard to judge CorticAll's performance. There is a "SOTA" column in the table that is blank on some handwriting and cursor datasets -- could the authors not try established baselines from the literature on these datasets so we understand the benefits of CorticAll in the context of prior work?
  • The architecture and/or preprocessing/tokenisation seem like they might be acausal, reducing the practical relevance -- please correct me if I've misunderstood! (Although I do understand speech is typically done in a full-utterance regime with some non-causal baselines).
  • The claim that this work is introducing "supervised on-scale pretraining" is a little oversold. Several prior works have trained jointly in a supervised fashion on several datasets, even if not with the breadth explored here, e.g., NDT-3 (Ye et al. 2025) trained a joint autoregressive model (so it's SSL+SL) on several tasks, POYO/POYO+ (Azabou et al. 2023/2025) and POSSM (Ryoo et al. 2025) have multitask readout to jointly train readouts which are relevant to the downstream task, etc. Furthermore, the authors only explore supervised pretraining here but recent work (Mao et al., 2026) has shown that joint SSL+SL training can enable this readout reuse downstream and also reduce labelled finetuning data requirements which the authors address, which seems relevant here.
  • An additional related point: a discussion of these works would be beneficial in this context, and especially given the claim that "SSL" is not as data-efficient as this supervised pretraining scheme: I could agree that SSL alone may not be stable downstream as readouts have to be learnt, but work like Ye et al. (2025) and Mao et al. (2026) do finetune well and efficiently downstream after "joint" pretraining. A claim about the data-efficiency of SSL could also be substantiated with a baseline -- currently missing.
  • It's unclear what the splits used in the dataset are, especially for hyperparameter tuning -- some datasets drop the test splits associated with benchmarks or competitions, and so it's unclear what the final numbers reported in the tables are and whether hyperparameter tuning was done on the same validation set that final numbers are reported on. This needs to be clarified.
  • Relatedly, checkpoint selection was also done based on the validation metric, but if it is not an independent val. and test set then the selection is biased and there should be an independent evaluation.
  • Minor: the terminology used is a bit confusing and non-standard. Typically, finetuning to my understanding is the term used when a dataset isn't necessarily part of the pretraining set but a session/dataset that is being generalised to. But over here it seems like the model may be trained on all datasets then specifically "continually" trained on just the target dataset. It's worth clarifying this in light of terminologies used in prior work.

References:

  1. POYO: Azabou et al. (2023). A unified, scalable framework for neural population decoding. NeurIPS 2023.
  2. POYO+: Azabou et al. (2025). Multi-session, multi-task neural decoding from distinct cell-types and brain regions. ICLR 2025.
  3. POSSM: Ryoo et al. (2025). Generalizable, real-time neural decoding with hybrid state-space models. NeurIPS 2025.
  4. NDT3: Ye et al. (2025). A generalist intracortical motor decoder. NeurIPS 2025.
  5. MOJO: Mao et al. (2026). Leveraging unlabelled data for generalizable neural population decoding. NeurIPS 2026.
Questions for authors

Clarifying questions have been included in the weaknesses section, but to summarise:

  • Why is all-modality training often worse than within-modality? Does it not go against the motivation of the work? And have the authors tried data selection, more tuning, etc.?
  • Do the authors have baselines for datasets denoted with "–" in the table for "SOTA"?
  • Can the authors provide evidence for the SSL data inefficiency statement? Can the authors also discuss this in light of prior work mentioned?
  • What are the splits and data subsets used for hyperparameter tuning and are they different from the data on which final numbers are reported?

Regarding my rating: if the above clarifications are addressed I'm happy to lean towards acceptance because I appreciate the effort to build a unified model, but want to ensure that evaluations are done correctly.

8 comments

We thank the reviewer for the detailed and constructive review. We address each point in turn.

  1. All-modality versus within-modality training

CorticAll shows that four communication modalities can be decoded with one set of weights, to our knowledge the first such demonstration for communication BCIs. This ability comes at a modest, modality-dependent cost relative to within-modality joint training. The change in modality-mean error at the final checkpoint is +1.2 points for speech [0.9, 1.5], +1.5 for handwriting [1.1, 1.8] and −1.1 for typing [−1.6, −0.5] (95% intervals), and the all-modality model gives the best initialisation for in-pool fine-tuning on handwriting and typing. The new analyses show where the differences come from (Figure 3, App. A.9):

The speech cost falls on the largest datasets (Card, Willett). The smallest, Kunz T16, improves by 8.6 points, because the other modalities regularise it: speech-only training memorises its 365 trials. Training modalities in pairs (table below) shows that the smaller modalities, typing in particular, improve when trained with data-rich speech, while speech itself is unchanged or slightly worse when a smaller modality is added. This mirrors multilingual machine translation, where sharing one model helps low-resource languages at a small cost to high-resource ones (Arivazhagan et al., 2019). Modality gradients are positively aligned early in training and unrelated later, without sustained opposition, so the cost does not come from conflicting objectives.

Change in error (points, 95% interval) when one modality is added to the target's training:

Target+ Speech+ Handwriting+ Typing
Speech–−0.1 [−0.4, +0.2]+0.5 [+0.1, +0.9]
Handwriting−0.6 [−1.0, −0.2]–+0.2 [−0.2, +0.5]
Typing−1.7 [−2.3, −1.0]−0.3 [−1.0, +0.4]–

We did not tune hyperparameters specifically for the multimodal setting. Per-modality loss weights and model size are the natural levers; exploring them was beyond the compute available for this rebuttal.

  1. Fine-tuning on unseen data; no non-speech leave-out

CorticAll was designed primarily for scalable joint training, and we approached transfer to unseen data as a test of whether the current data regime already supports foundation-model-style supervised pretraining. The earlier experiment was replaced with a leave-one-participant-out study (Table 2, Figure 4). At full target data, fine-tuning is better than training from scratch on Wairagkar (−4.5 PER) and Jude (−2.8), ties on Kunz T15 and T17, and is worse on Card (+0.8) and on the T12 and T16 datasets (+2.8 to +7.7). With less data, the minutes of target data needed to reach a given PER are (from scratch / fine-tuned, interpolated from the learning curves):

PERCard T15Willett T12Jude T17
60%121 / 5022 / 4293 / 111
50%183 / 8651 / 70243 / 191
40%280 / 153120 / 120441 / 323

Pretraining roughly halves the data T15 needs, helps T17 at lower error levels, and does not help T12. One reading is that the present corpus covers some, but not all, of the patterns a new participant brings; with four speech participants this remains a hypothesis. LOPO is limited to speech because every held-out participant needs its own pretraining run (compute in App. A.6); running it on the all-modality model was beyond our budget, and speech has the most data and participants.

  1. Baselines for datasets without a published comparator

The baseline keeps our pipeline and Temporal Transformer and replaces only the front-end with the dataset-specific linear patch embedder of BIT, the architecture behind the published state of the art on the other speech datasets. It is therefore a strong reference that isolates the effect of the shared front-end. Both models are trained from scratch on the single dataset and fine-tuned from an all-modality model (one seed for the baseline):

DatasetBaseline, scratchCorticAll, scratchBaseline, fine-tunedCorticAll, in-pool fine-tuned
Kunz T16 (PER)66.224.225.421.8
Kunz T17 (PER)34.635.937.431.9
Jude T18 handwriting (CER)14.79.12.02.3

For cursor we chose not to add a baseline: the targets are themselves the output of the source studies' decoders, so small differences in R² between downstream models would not be conclusive. The cursor results in Table 1 are meant to show that CorticAll reproduces the decoded cursor trajectories well (R² up to 0.93) while serving the other three modalities.

  1. Causality

The reading is correct: the model is non-causal (bidirectional transformer, per-trial mixing weights, per-session normalisation), now stated in both Section 2.2 and the limitations. The BIT results we compare with on speech also come from a bidirectional, full-utterance model. Online decoding is a separate design problem that we leave to future work.

  1. "Supervised-on-scale" and related work

We agree that prior work on supervised multi-dataset training should be acknowledged, and have expanded the Introduction and Discussion to discuss POYO, POYO+, POSSM, NDT3 and MOJO. CorticAll addresses a related, yet notably different problem. POYO and NDT3 focus primarily on continuous motor decoding with time-aligned targets; POYO+ extends this to multiple tasks and brain regions, and MOJO adds a self-supervised objective. NDT3 handles variable channel counts by fixed-size patching without channel identity, and its authors report that the resulting sensitivity to input ordering limits cross-subject transfer. POSSM additionally evaluates human handwriting (individual characters) and speech, but with separate models for these tasks rather than decoding them jointly. In contrast, our speech, handwriting and typing tasks require predicting variable-length sequences of phonemes or characters without known temporal alignment. CorticAll trains these three tasks jointly, alongside continuous cursor control, across heterogeneous human intracortical recordings, with a shared backbone and modality-specific heads. To our knowledge, joint decoding of these four communication modalities in one model has not been demonstrated before.

  1. Self-supervised pretraining

We removed the data-efficiency claim. The paper motivates supervised training by two observations: recent supervised cross-subject speech decoders report clear gains, and public intracortical communication datasets are all released with labels, which makes them directly usable for supervised pooling. We describe self-supervised and joint objectives as complementary. App. A.11 reports the preliminary comparison that informed our choice of the supervised route, using an earlier CorticAll variant differing only in its per-channel temporal block. Card validation PER (%) by fine-tuning epoch:

Initialisation50100200400
From scratch12.910.59.58.5
Supervised, all modalities9.38.98.67.9
Supervised, speech without Card10.69.78.87.8
Self-supervised (masked reconstruction)15.613.412.211.5

We state its limits (one recipe, one seed, one data-rich target, no joint objective) and do not claim that supervision is preferable in general.

  1. Splits, tuning and checkpoint selection

Splits are listed per dataset in App. A.1, with their relation to the original studies. There is no separate test set: the Brain-to-Text test labels are not public, and those benchmarks are scored at the word level (WER after a language model), not at the phoneme level, so all results are on validation splits. We did not run a hyperparameter sweep. During development we performed limited manual tuning, mainly of the Channel Mixer, using the validation metric aggregated over all datasets. The Temporal Transformer's stack and hyperparameters were retained from Feghhi et al. All results now use the final checkpoint, so the reported numbers are independent of checkpoint selection. Modality means differ from the best-checkpoint values by at most 0.9 points (Table 12).

  1. Terminology

We use "fine-tuning" in the general sense of continued training of a pretrained model and qualify it throughout: in-pool fine-tuning (regime 4, target dataset in the pretraining pool) and LOPO fine-tuning (regime 5, participant absent from pretraining). They are separate regimes in Section 2.3 and separate tables (Table 1 and Table 2).

Reference: Arivazhagan et al. (2019). Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges. arXiv:1907.05019.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer,

As the discussion period is approaching its deadline, we would appreciate any feedback on our rebuttal, particularly whether it addresses your concerns. We would be happy to clarify any remaining points.

Thank you, Authors

Reviewer_YwEkOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CorticAll, a unified intracortical BCI decoder designed to handle heterogeneous neural recordings across participants and multiple communication modalities, including speech, handwriting, typing, and cursor control. The key architectural contribution is a shared channel tokenizer, learnable channel identity embeddings, and a cross-attention channel mixer that maps varying numbers of neural channels into a fixed-dimensional representation. This allows the model to share most of its parameters across datasets while preserving channel-specific information.

The authors evaluate CorticAll on 11 datasets from 7 participants, covering approximately 173 hours of neural recordings. They investigate within-modality and multimodal supervised training, target-specific fine-tuning, and transfer to held-out datasets. The results show that pooling data across participants generally improves decoding performance, and fine-tuning the pooled model achieves better error rates than published baselines on 7 of 8 datasets with available comparisons. Additional ablation studies demonstrate the importance of channel identity embeddings and the shared frontend, while latent-space analyses suggest substantial shared representations across participants and modalities.

Critical strengths
  1. A well-motivated and effective architecture for heterogeneous neural inputs. I like how the proposed architecture directly addresses the challenge of combining neural recordings with different channel counts and identities. The combination of a shared channel tokenizer, channel identity embeddings, and a cross-attention channel mixer is intuitive and practically useful. The ablation results are particularly convincing. Removing channel identity embeddings leads to a dramatic performance drop across all four modalities, while the shared frontend also outperforms the BiT-style dataset-specific frontend with fewer parameters (14.3M vs. 23.0M). These results provide concrete evidence that the proposed architectural design is effective.

  2. Strong empirical results across diverse datasets and training settings. I appreciate the extensive evaluation across 11 datasets, 7 participants, and four modalities. The improvements from pooling data across participants are particularly encouraging, especially for smaller datasets. For example, within-modality training reduces the speech PER from 67.4% to 32.4% on Kunz T17 and the handwriting CER from 8.5% to 2.4% on Jude T18. I also find the leave-one-dataset-out experiments valuable, as they show that the pretrained model can improve performance on held-out speech datasets after fine-tuning. These results demonstrate the practical benefits of learning from heterogeneous neural data.

Critical weaknesses

The benefits of multimodal training are not consistently demonstrated. While the paper shows clear improvements from pooling data across participants within the same modality, I am less convinced about the additional benefits of training across different modalities. In Table 1, all-modality training actually performs worse than within-modality training for speech (13.8% vs. 12.8% PER) and handwriting (11.8% vs. 10.7% CER), although it improves typing performance. Even after fine-tuning, the benefits remain modality-dependent. Since multimodal learning is one of the paper's central contributions, I would like to better understand when and why combining different modalities helps, beyond simply demonstrating that a single model can handle them.

Questions for authors
  1. Generalization to unseen participants. In the leave-one-dataset-out experiments, some participants still appear in the pretraining data through other datasets. Could the authors clarify how well the model generalizes to entirely unseen participants? In particular, Kunz T16 appears to be an unseen participant in this setting, but its PER increases from 23.0% when trained from scratch to 40.4% after transfer. I would be interested in understanding why transfer performs substantially worse in this case and whether the authors have additional evidence about generalization to new participants.

  2. Benefits of cross-modality training. Table 1 shows that all-modality training improves typing performance but performs worse than within-modality training for speech and handwriting. Could the authors provide more insight into why different modalities benefit differently from joint training? In particular, is there evidence of negative transfer between certain modalities, and what determines when cross-modality training is beneficial?

4 comments

We thank the reviewer for the positive and careful assessment. Both questions are addressed with new experiments; the key numbers are given here.

  1. Generalisation to unseen participants, and Kunz T16

CorticAll was designed for joint training across participants and modalities; transfer to unseen participants is an additional test of whether supervised pooling can already act as foundation-model-style pretraining. We replaced the leave-one-dataset-out experiment with leave-one-participant-out (LOPO) pretraining for all four speech participants, so every target is a truly unseen participant.

PER (%) at full target data, by which parameters are trained:

TrainedCard T15Wairagkar T15Kunz T15Willett T12Kunz T12Kunz T16Jude T17Kunz T17
None (zero-shot)83858480858279195
Embeddings only66.666.781.879.492.180.494.460.7
Front-end (1.3M)13.512.125.726.448.031.035.632.4
All (14.1M)9.58.330.417.334.731.732.233.5
From scratch8.612.727.014.626.924.235.035.9

A new participant always needs calibration: zero-shot outputs are unrelated to the trial, and learning the new channel embeddings alone is not enough. With fine-tuning, the result depends on the participant. Pretraining helps T15 and T17 when target data are limited (learning curves in Figure 4) and on Wairagkar and Jude at full data, and has no positive effect for T12 and T16.

For Kunz T16 specifically: the earlier value (40.4) came from a fine-tuning recipe that was unstable for new participants. With the recipe selected in the revision T16 reaches 31.7% PER against 24.2% from scratch, so pretraining without T16 still does not help it. T16 is the most atypical participant in the corpus: pontine stroke with residual dysarthric speech, one modelled 64-channel array, the smallest dataset, and the lowest shared latent variance (Figure 8). One reading of the participant dependence is that the present corpus covers some, but not all, of the neural patterns a new participant brings, so transfer succeeds where they are covered. With four speech participants this remains a hypothesis; App. A.10 lists the factors we could rule out.

  1. When and why does cross-modal training help? Is there negative transfer? Modality means at the final checkpoint (error in %, cursor in R²):
Within-modalityAll modalitiesIn-pool fine-tuned from withinIn-pool fine-tuned from all
Speech (PER)12.814.011.812.4
Handwriting (CER)10.812.310.710.3
Typing (CER)15.514.414.913.1
Cursor (R²)0.600.570.620.63

To see which modalities help or hinder each other we trained models on pairs of modalities. Change in error (points, 95% interval) when a modality is added to the target's training:

Target+ Speech+ Handwriting+ Typing
Speech–−0.1 [−0.4, +0.2]+0.5 [+0.1, +0.9]
Handwriting−0.6 [−1.0, −0.2]–+0.2 [−0.2, +0.5]
Typing−1.7 [−2.3, −1.0]−0.3 [−1.0, +0.4]–

Cross-modal training helps on small datasets, as a regulariser: trained on speech alone, the model memorises the 365 Kunz T16 training trials and validation error rises by 5.4 points after its minimum; with all modalities this is largely prevented and T16 improves by 8.6 points. Typing gains from co-training with speech, the data-rich modality. It costs the largest speech datasets (Card, Willett) about 1 to 2 points, and handwriting in the all-modality model. This pattern partially mirrors multilingual machine translation, where sharing one model helps low-resource languages at a small cost to high-resource ones (Arivazhagan et al., 2019).

On negative transfer, we measured gradient alignment between modalities during training. All pairs are positively aligned early and become unrelated later, without sustained opposition, so the costs do not come from conflicting objectives. Loss balancing and data mixture, which we did not tune, are the natural levers.

In short, a single set of weights decodes four modalities at a modest cost, and gives the best initialisation for in-pool fine-tuning on handwriting and typing.

Reference: Arivazhagan et al. (2019). Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges. arXiv:1907.05019.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer,

As the discussion period is approaching its deadline, we would appreciate any feedback on our rebuttal, particularly whether it addresses your concerns. We would be happy to clarify any remaining points.

Thank you, Authors

Thank you for the detailed response and additional experiments. I appreciate the new leave-one-participant-out experiments, which address my concern about evaluating transfer to entirely unseen participants. Although the results still show limited improvements for some participants, particularly T16, they provide a clearer picture of the model's generalization ability.

I also appreciate the pairwise modality experiments. They help explain why multimodal training benefits some tasks but not others. I think these additional results strengthen the paper and address my main questions.

Overall, I remain positive about the paper and plan to maintain my rating of 3 (Weak Acceptance).

Thank you for going through the new experiments and for the positive assessment.

Comments on the paper

3 comments

Dear Reviewer,

Thank you very much for your time and thoughtful review. As the discussion deadline approaches, could you please take a look at the authors' rebuttal and share your thoughts in the discussion?

Many thanks for your continued help!

AuthorsOpenReview ↗

General response to all reviewers and the AC

We thank all four reviewers for their careful reading. CorticAll set out to show that one decoder can pool intracortical recordings across participants, cortical areas and four communication modalities, and to study when such pooling helps. We are glad the reviewers recognised its main strengths: a well-motivated architecture for heterogeneous inputs backed by convincing ablations (YwEk), a novel multimodal training setup on a corpus that aggregates the publicly available intracortical communication datasets (XWoF, k7Bc), a numerical improvement on the published state of the art on 7 of 8 datasets (YwEk, k7Bc), and a well-substantiated latent-structure analysis (95KV). The reviews also pointed to analyses that would strengthen the work. We carried out those feasible within the rebuttal period, and believe they make the paper considerably stronger. Text that is new or changed relative to the original submission is shown in blue in the revised PDF.

New experiments and analyses:

  1. Evaluation independent of checkpoint selection, with confidence intervals (k7Bc, XWoF): all results are reported at the final checkpoint (Table 1), with 95% bootstrap intervals for every result and paired differences between regimes (App. A.7).
  2. Transfer to participants unseen in pretraining (95KV, XWoF, YwEk, k7Bc): leave-one-participant-out (LOPO) pretraining for all four speech participants, comparing zero-shot use, embeddings-only, front-end-only and full fine-tuning (Table 2), and learning curves from 5 minutes to 8 hours of target data against training from scratch (Figure 4, App. A.10).
  3. Interactions between modalities (YwEk, k7Bc): models trained on pairs of modalities, training dynamics on small datasets, and gradient alignment between modalities (Figure 3, App. A.9).
  4. Baselines for datasets without a published comparator (k7Bc), in App. A.8.
  5. A preliminary comparison with self-supervised pretraining (k7Bc), in App. A.11.

On framing: the paper is framed as a study of when additional data helps (same modality, other modalities, other participants), using an architecture built for pooling. "Supervised-on-scale" names the regime we study. We credit POYO, POYO+, POSSM, NDT3, MOJO and Fogg et al., and state the lineage of our channel mixing. "Universal" is defined in the Introduction: any configuration of intracortical arrays and channel sets, and all four major BCI communication modalities in one model. The claim that supervised pretraining is more data-efficient than self-supervised pretraining was removed.

Together, the new experiments show where pooling helps most, what a single multimodal model costs, and what a new participant requires.

AuthorsOpenReview ↗

Summary of the discussion (for the AC and all reviewers)

We thank the reviewers and the AC for the discussion. As it closes, we summarise what was requested and where each point is addressed in the revised PDF (changes in blue), so that all four reviews can be read against the revision. Two analyses were added after our general response and are not listed there: a check on the hidden Brain-to-Text test sets (App. A.7) and an analysis of online use (App. A.12).

Reviewer k7Bc (rating raised from 2 to 3 during the discussion; the reviewer has stated that they support acceptance) asked for an evaluation that does not depend on checkpoint selection or tuning, baselines for datasets without a published comparator, a clearer positioning against POYO, POYO+, POSSM, NDT3 and MOJO, evidence for the claim about self-supervised pretraining, and clarification of causality.

  • Final-checkpoint results (Table 1) and a check on the hidden Brain-to-Text test sets (App. A.7).
  • Internal baselines (App. A.8); revised related work (Introduction); the data-efficiency claim removed and a preliminary comparison reported (App. A.11).
  • An analysis of online use without retraining (App. A.12).

Reviewer YwEk (rating 3, maintained; the reviewer has indicated that the additional results address their main questions) asked how the model generalises to unseen participants and when cross-modal training helps.

  • Leave-one-participant-out experiments for all four speech participants, including which parameters must adapt and how the benefit depends on target data (Table 2, Figure 4, App. A.10).
  • Pairwise modality co-training, the regularising effect on small datasets, and gradient alignment between modalities (Figure 3, App. A.9).

Reviewer 95KV (rating 2) asked for learning curves over target-data quantity, for clarity on which parameters are updated (including no fine-tuning and embeddings only), and for the channel-embedding ablation to be related to the design of Fogg et al.

  • Learning curves from 5 minutes to 8 hours against training from scratch (Figure 4, App. A.10).
  • Zero-shot, embeddings-only, front-end-only and full fine-tuning for eight speech datasets (Table 2).
  • The comparison with a per-dataset input projection, the design used by Fogg et al., and a discussion of that work (App. A.2, Introduction, Discussion).

Reviewer XWoF (rating 2) asked for confidence bounds on the comparison with published results, questioned the cursor targets, and asked what "universal" means and what a new participant requires.

  • Bootstrap intervals for every result and for the comparison with published values (App. A.7).
  • The purpose of the cursor modality stated, and its R² removed from the abstract (Section 2.1).
  • "Universal" defined where it is first used; the requirements for a new participant quantified (Table 2, Figure 4).

Not addressed. Larger validation sets are not possible, since the validation splits are fixed by the source datasets. Additional seeds and a leave-out study outside speech were not feasible within the rebuttal period: each requires retraining across datasets and regimes, at a compute cost documented in App. A.6. A joint self-supervised and supervised objective and a causal variant of the model are left to future work.

We are grateful for the reviewers' time and remain available for any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.