- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Authors are introducing CREWS, a new approach to generalize speech BCI models across subject in a (quasi) zero-shot fashion. Instead of relying on subject-specific read-outs they train a region specific encoder which has parameters for each region of the broadmann atlas in which electrodes could be implanted in the pre-motor cortex for attempted speech task. They use neural recordings coming from BrainGate2 studies, namely the Card et al, Willet et al and part of the Kunz et al datasets, all of which carry 1 participant implanted with multiple Utah arrays, sometimes with the same region overlapping across participants (in particular area 6v).
They proceed evaluating the model by training it on Card and Willett data using the baseline backbone GRU proposed earlier but replacing the subject and session specific layers with their shared per region encoder, averaging the activity over each region, showing sparks of generalization across subjects measuring the PER on held-out T16 subject (and rotating them) with zero or up to 60 calibration trials. Their main claim is that this approach is able to generalize across subject.
Critical strengths
There are a couple of things that I liked a lot:
-
The overall idea of CREWS, moving from subject specific to region specific across subject. Clear and simple intuition.
-
The way authors use the neural data statistic to reorder channels within each array (and therefore region). I found it clever, a sort of functional alignment done based on ranking, cool.
-
The problem they want to target itself, which is today one of the major limitation in deployment of effective speech BCI.
Critical weaknesses
While the idea of the paper is cool, there are some weaknesses that I'd like to list.
- The most obvious one, and I understood it's not fault of the authors, is the limited in scope analysis on new subject. Data are extremely limited, and in the end, there is only one subject avaialble for generalization and 1-2 for training, with just one region overlapping. Claims of the paper I think overstate the claims given the narrower scope of the current analysis. These are mostly all the publicly avaialble data of this kind, so I see where this come from, but still limits the results. At least, I would suggest authors to include data from Kunz et al 2025 T12 and T15 and T17 participants. In case of T12 and T15 these are the same subjects of the Willett and Card et al papers used for training, but in a different setting: imagery speech vs attempted speech. We already know that imagery and attempted speech exhibit similar structure in the neural data (but a motor intent direction) and showing generalization across the same regions (because of same participants) but different task will strengthen the results of the paper. T17 comes in a similar fashion (imagined single words) and could help substantiate the message if there is at least an array implanted in shared region.
- I am not convinced that CREWS is decoding phonetic content in the zero-shot setting, or at least the current evidence does not show it clearly. CTC is prone to degenerate solutions, and a PER of ~78% leaves a lot of room for non-phonetic explanations. The trial-shuffle chance level holds the decoded sequences fixed and permutes the references. It therefore controls for the overall distribution of output lengths and for predicting frequent phonemes, but not for the trial-wise correlation between output length and reference length. Trial duration correlates strongly with the number of phonemes and words in a sentence, so a decoder whose output length simply tracks trial duration would already beat this chance level without capturing phonetic structure. A substantial part of the gap to chance (91.4 to 77.7) could stem from this length information alone. The zero-shot claim I think should be toned down across the paper.
I acknowledge that the within-array electrode shuffle in Fig. 2 raises PER to chance, which partially argues against a pure length or energy explanation. However, that manipulation also changes the input statistics and may simply miscalibrate the decoder, so it is not conclusive. I would ask the authors for direct controls: a length matched chance level (permuting references only among sentences of similar length), a “length oracle” baseline that emits frequent phonemes at the correct length, a substitution/insertion/deletion breakdown, and the hypothesis-to-reference length ratio. On the closed 50-word vocabulary, word identification accuracy or WER with a 50-word language model would also be far more interpretable than PER. Unless these controls clearly support phonetic decoding, I would recommend that the zero-shot claims be substantially toned down throughout the paper, including the abstract and the contribution list.
- At the end of the day, qith the current cohort, only v6v is shared across all subjects, and every target-subject result is obtained through the v6v slot alone (T16 is decoded exclusively from its v6v array). The evidence therefore supports a narrower claim than the one made: that v6v can be modelled across subjects with a shared encoder. Nothing is shown about transfer through the other regions or about the multi-region slot design as such, and the authors acknowledge this in the limitations. The method combines two distinct ingredients: (i) the shared region-specific front-end with slot projections and averaging, and (ii) the label-free reordering of channels within each array by autocorrelation rank. The paper does not clearly separate their contributions.
The “v6v slot only” ablation in Table 4 is informative here. Restricting every patient to the v6v slot gives a zero-shot PER of 79.0, essentially the same as full CREWS (which was 77.7), and the difference appears only in the few shot regime. On the other hand, removing the autocorrelation ordering barely affects zero-shot PER (80.7) but severely impairs adaptation (50.1 vs. 23.5 at k = 60). Taken together, this suggests that the channel remapping may be the key ingredient and the multi-region front-end a secondary one. To disentangle the two, I would ask for: (a) the shared-input-layer baseline (all arrays, no region slots) with the autocorrelation reordering applied; (b) the patient-specific (Levin-style) baseline restricted to the v6v array, with and without reordering; and (c) a model trained only on the v6v arrays of T12 and/or T15 with reordering, applied to T16 zero-shot and after k-sentence calibration. If these variants match CREWS, the contribution of the shared multi-region encoder would be smaller than claimed. The value of the remapping procedure, however, would come out more clearly, and in my view it is the most interesting and reusable idea of the paper.
Questions for authors
-
See weakness 1, could you add more data to showcase cross-task benefit of this approach?
-
See weakness 2, could you provide example of decoded sentences at every level of K? Is the zero-shot merely matching the sequence length or it already capture some phonetic structure? Is it just predicting frequent phones? On the same line, since CTC might be unstable, would you be able to provide seed-by-seed variability of performance over K? the curve is not always monotic, even though they all converge stably at K=60.
-
Please see weakness 3
-
It's a bit unclear which data were used T16. It's attempted speech or imagined speech? Isolated words or full sentences? Have you checked for repetition/leakage of train/test data? Often sentences in Kunz dataset are repeated and a random split might allow for some unwanted leakage.
-
What does the autocorrelation ordering actually align? The autocorrelation of threshold crossings and spike-band power at 40–100 ms lags is likely to covary strongly with firing rate and signal-to-noise ratio. The ordering may therefore sort electrodes by recording quality (“good channels first”) rather than by any functional correspondence across patients, and the shared encoder would then learn a quality-weighted readout. Could the authors compare the autocorrelation order against simple orderings by mean firing rate, SNR, or variance? If these yield similar zero-shot and few-shot PER, the mechanism behind the remapping would need to be reframed. Relatedly, how stable is the ranking within a subject across sessions (test-retest Spearman), and how much unlabeled data is needed before the order, and the downstream PER, stabilizes?
-
Is there a shared latent geometry, or is the GRU compensating? You are arguing that region slots create corresponandance across patients, but it was never shown at representations levels. Could you compare the 6v encoder outputs across T12, T15 and T16 for matched sentences or words or phonemes? I would suggest to use RSA, CCA or Procrstes alignment testing with and without the autocorrelation ordering, versus the normal baseline read in layer. It would be nice to see whether the model is able to project different subjects in a common geometric space.
5 comments
Response to Reviewer doFQ (2/3): W3·Q3, Q4
W3 · Q3 (separating the autocorrelation ordering from the region front-end). We trained the three requested variants (a)–(c) and added a model that uses only the v6v slot without the autocorrelation ordering as a control. CREWS with and without the autocorrelation ordering is listed in the first two rows for comparison.
| Variant (PER) | Input | Ordering | Zero-shot (chance) | k=5 | k=10 | k=20 | k=60 |
|---|---|---|---|---|---|---|---|
| CREWS | all region slots | yes | 77.7 (91.4) | 68.9 | 61.5 | 45.2 | 23.5 |
| CREWS, no ordering | all region slots | no | 80.7 | 77.4 | 72.3 | 67.0 | 50.1 |
| (a) | shared input layer | yes | 99.6 (99.5) | 74.8 | 57.1 | 46.2 | 28.0 |
| (b) | patient-specific layer, v6v | either | – | 71.9 | 68.9 | 70.8 | 25.1 |
| (c) | v6v slot only | yes | 79.0 | 77.3 | 69.1 | 57.4 | 28.4 |
| Control | v6v slot only | no | 79.1 (96.9) | 78.3 | 67.2 | 67.8 | 66.2 |
Ordering denotes the autocorrelation ordering. The first, second, and (c) rows are from Tables 1 and 4. The zero-shot entry of (b) is empty because the model produces no output without T16 labels.
(a) On T16, the zero-shot PER of the shared input layer is at the chance level with or without the autocorrelation ordering. Zero-shot transfer therefore requires the region slots. In adaptation, variant (a) attains a lower PER than the shared input layer without the autocorrelation ordering at k=5, 10, and 20 (Table 1). As suggested, the autocorrelation ordering can therefore be reused independently, although its benefit is smaller without region slots.
(b) For the patient-specific input layer, the model with the autocorrelation ordering and the model without the autocorrelation ordering are the same model. The autocorrelation ordering only permutes the columns of that layer, and the logits of the two models differ by less than 1e-7.
(c) With only the v6v slot, removing the autocorrelation ordering leaves the zero-shot PER almost unchanged, whereas adaptation degrades at k=20 and k=60 (rows (c) and Control). CREWS exhibits the same pattern, as the first two rows of the table show.
In summary, the region slots account for zero-shot transfer and the autocorrelation ordering accounts for adaptation. The effect of the autocorrelation ordering is absent for patient-specific input layers, small for the shared input layer, and largest in combination with the region slots. Neither component substitutes for the other, and this distinction will be stated in Sec. 5.4 of the revision.
Q4 (T16 data). The T16 data are attempted-speech sentences from a single session. No sentence is repeated, and the overlap between adaptation and evaluation sentences is 0. The zero-shot PER of CREWS is unchanged on the 75 sentences whose transcript does not appear in the T15 training set (table in the General Response, G2). The Setup of the submission did not describe these conditions in sufficient detail. The Setup of the revision will state the task (attempted speech, sentences), the number of sessions, and the absence of repeated sentences, and the appendix will report the overlap with the T15 training sentences.
Response to Reviewer doFQ (3/3): Q5, Q6
Q5 (what the autocorrelation order aligns). To examine whether the order ranks electrodes by recording quality, we replaced only the ordering key with the firing rate, the variance, or the SNR and repeated training from the source stage.
| Training with a different ordering key (PER, %) | Zero-shot | k=5 | k=10 | k=20 | k=60 |
|---|---|---|---|---|---|
| Firing rate | 83.5 | 80.1 | 74.3 | 47.4 | 26.3 |
| Variance | 90.4 | 75.5 | 65.0 | 53.8 | 60.5 |
| SNR | 82.9 | 77.2 | 56.5 | 42.3 | 27.8 |
| Autocorrelation (CREWS, Table 1) | 77.7 | 68.9 | 61.5 | 45.2 | 23.5 |
The autocorrelation ordering yields the lowest PER in the zero-shot setting and at k=5 and k=60. The autocorrelation ordering therefore captures more than a ranking by recording quality.
We also examined the stability of the order across recording days and the amount of unlabeled data that the order requires. The order of T16 was computed without labels from its 60 training trials, which were recorded in a single session. Reproducibility across recording days is therefore reported for T15 and T12. Across recording days, the median Spearman correlation of the v6v order is 0.68 for T12 and 0.31 for T15. The order can therefore differ across recording days, and this point will be stated in the Limitations of the revision. The required amount of unlabeled data is reported in the following table. For the downstream PER, CREWS was adapted on k sentences of T16 with the order computed from the same k trials.
| Number of unlabeled trials | 5 | 10 | 20 | 50 | 60 |
|---|---|---|---|---|---|
| T15, Spearman with the order from all trials (v6v) | 0.69 | 0.75 | 0.87 | 0.93 | 0.94 |
| T12, same measure | 0.88 | 0.92 | 0.96 | 0.98 | 0.98 |
| T16, Spearman with the order from the 60 training trials | 0.94 | 0.97 | 0.99 | 1.00 | 1.00 |
| T16 downstream PER (%) | 77.9 | 72.0 | 47.1 | 31.8 | 23.5 |
| CREWS, order from the 60 training trials | 68.9 | 61.5 | 45.2 | 30.7 | 23.5 |
For T12 and T16, the order itself converges with 20 unlabeled trials (ρ ≥ 0.96 with the reference order). For T16, the downstream PER follows a similar trend: with 20 or more trials, the order from the k adaptation trials yields a PER close to that of the order from the 60 training trials.
Q6 (shared latent geometry). Following the suggestion, we compared the output of the v6v region encoder of the three patients using phoneme-level RSA and Procrustes alignment. Top-1 denotes the proportion of phonemes whose nearest prototype in the other patient, after Procrustes alignment, is the same phoneme (chance approximately 0.03).
| v6v region encoder output | RSA T15–T12 | RSA T16–T15 | RSA T16–T12 | Top-1 Procrustes T15–T12 |
|---|---|---|---|---|
| CREWS | 0.41 | 0.21 | 0.18 | 0.28 |
| No electrode autocorrelation ordering | 0.22 | 0.11 | 0.06 | 0.11 |
| Shared input layer | 0.25 | 0.09 | 0.01 | 0.08 |
The region encoder outputs of CREWS share the phoneme similarity structure across all three patient pairs, including T16, which was not used for training. All four values are higher than those of the model without the autocorrelation ordering and of the shared input layer, indicating that the autocorrelation ordering strengthens this shared structure when combined with the region slots. A structure shared across patients is therefore already present at the region encoder output, before the GRU. This analysis will be included in the appendix of the revision.
We are grateful for the detailed suggestions, which led to most of the new analyses in the revision. We would be glad to provide any further control that would be useful.
Response to Reviewer doFQ (1/3): W1·Q1, W2, Q2
We thank the reviewer for the thoughtful assessment and feedback. We performed the requested experiments and respond point by point.
W1 · Q1 (data of Kunz et al., 2025; T17). As suggested, we evaluated CREWS on the data of Kunz et al. (2025), and these results will be added to the revision. None of the sentence evaluations used a model trained on imagined speech. For T17, whose data contain only isolated words, CREWS was adapted on 10 imagined trials per word.
In every evaluation of the table, the PER of CREWS is clearly below the chance level. These results indicate that CREWS also applies to other tasks. No evaluation used patient-specific parameters.
W2 (whether zero-shot decoding reflects phonetic content). We added all requested controls (General Response, G1), and these controls show that the zero-shot result is not explained by length.
The length-matched chance level (90.9%) and the length oracle (89.62%) both remain far above the PER of zero-shot CREWS (77.7%). The error breakdown and the hypothesis-to-reference length ratio are listed in the following table. Substitutions are the most frequent error, and the zero-shot output is neither empty nor far shorter than the reference sentence.
Following the suggestion, the WER on the 50-word-vocabulary sentences of T16 was computed using a language model. The language model is the 3-gram released by Willett et al. (2023), and decoding used the same settings as the evaluation of T12. Word identification accuracy denotes the proportion of the words in the reference sentence that appear in the decoded output. The values in parentheses were obtained by permuting the reference sentences among trials of similar length.
The WER is below the permuted value already in the zero-shot setting, and the margin increases with the number of adaptation sentences.
Regarding the suggestion to tone down the zero-shot claim, the requested controls indicate that the zero-shot result reflects phonetic content rather than length. We therefore retain the claim that zero-shot decoding is below the chance level. However, we agree that the phrase "viable zero-shot decoding" in the abstract can be read as a claim of accuracy at a usable level. We will remove "viable" in the revision so that the abstract matches the result.
Q2 (decoded examples at every k, frequent phonemes, seed variability). Decoded examples at every k for three evaluation sentences are shown in the following table and will be included in the appendix of the revision.
The zero-shot output is not restricted to frequent phonemes, as the token-type count and the length oracle in the General Response (G1) show.
Across three seeds, the standard deviation of CREWS lies between 1.2% and 1.5% at every k. For comparison, the following table lists two baselines that add a new patient-specific input layer for T16 to a source model. The frozen decoder (Table 1) trains only this new layer. The fine-tuned baseline, which is not part of Table 1, additionally updates all other parameters, using only the k sentences of T16. Both baselines yield a higher PER and a larger standard deviation than CREWS at every k.