acceptodds
Under review as a conference paper at ICLR 2027

Removing Brainless Shortcuts Improves Non-Invasive Brain-to-Text

10 views

Abstract

We find that major reported improvements in decoding words from non-invasive brain recordings are largely reproducible without any brain data. In the influential work of d'Ascoli et al. (2025), time series of brain activity from subjects perceiving continuous speech are segmented into fixed-length windows starting at each word. A neural network then generates predictions for all of the words in a sentence together. Neighbouring windows partially overlap, implicitly revealing the interval between words. Since these intervals indicate the duration of the words spoken, and different words tend to take longer – for example, "the" is much shorter than "supercalifragilisticexpialidocious" – the neural network can improve its predictions of words without relying on the underlying brain activity. Consistent with this, the method reaches 22.0% balanced accuracy on synthetic signals containing no brain information, compared with 22.3% on real brain recordings. To prevent the network from learning this shortcut, we make a single, simple change. Instead of jointly encoding all windows in a sentence, we process each independently. As a result, the neural network achieves better performance by learning underlying word-specific information from brain recordings. This makes two existing strategies become much more effective than before. Both aggregating predictions from distinct neural responses to the same word and using a pretrained LLM as a linguistic prior now substantially improve results. On our clinically motivated perceived speech benchmark, this simple recipe (SimpleB2T) achieves a word error rate of 36.6% with five observations per word, approaching past invasive speech decoding performance, albeit under different conditions. The results in this work expose an important shortcut in brain-to-text decoding and show that removing it leads to a simple and considerably more effective strategy.

Primary area:
applications to neuroscience & cognitive science
Keywords:
brain-computer interfaces, neural decoding, speech decoding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nN56OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies a timing shortcut in word-aligned non-invasive brain-to-text decoding: overlapping MEG windows expose inter-word intervals that are informative about word identity. Synthetic signals preserving this overlap achieve 22.0% balanced word accuracy, compared with 22.3% on real MEG. The authors independently decode word windows and combine repeated observations with language-model rescoring, obtaining 36.6% WER with five observations per word on an assembled communication benchmark. Timing-only controls, retraining without overlap, and evaluation across additional subjects support the shortcut analysis.

Critical strengths

The synthetic-signal experiment directly demonstrates that substantial decoding accuracy can arise without neural content when overlap structure is preserved. The timing-only baseline and additional-subject evaluation strengthen the interpretation that inter-word timing is an exploitable source of information. This finding has practical consequences for constructing controls and interpreting progress in word-aligned decoding.

The corrective approach is simple, and the experiments show useful complementarity between neural predictions and linguistic priors. On the Core benchmark with five observations, combining the neural decoder and LM reduces WER to 36.6%, compared with 74.3% for neural predictions alone and 73.8% for the LM alone. Noise and shuffled-embedding controls further support dependence on correctly matched neural input.

Critical weaknesses

The headline sentence-reconstruction result is measured on sentences assembled from separately recorded word occurrences, with repetitions selected using reference word identities. This provides a controlled test of combining lexical evidence, but does not establish reconstruction of complete sentences actually perceived by the participant. Consequently, its relevance to practical communication and the comparison with invasive speech-decoding WER remain limited.

The comparison does not fully isolate independent decoding as the cause of improved reconstruction. Section 3.1 replaces an approximately 200M-parameter contextual transformer with a 20M-parameter CNN–MLP, changing both cross-word access and architecture. Appendix J usefully addresses overlap mismatch between training and evaluation, but differences in capacity, optimization, and decoder calibration remain possible explanations for part of the advantage.

Questions for authors
  1. What stimulus-identity checks support the statement in Section 2.3 that stimuli do not overlap across partitions? Appendix B specifies session-level separation, while LibriBrain100 documents repeated TIMIT sentence texts across speakers. Please provide transcript-level overlap counts and clarify whether separation concerns audio recordings, sentence texts, or both.

  2. Were temperature and LM weight selected independently for each decoder using the same development set? A comparison retaining the transformer architecture while disabling cross-word attention would help distinguish the effect of independent decoding from architectural changes.

  3. Does SimpleB2T improve sentence reconstruction on original held-out utterances at (k=1), under a stated vocabulary and out-of-vocabulary protocol? This would clarify whether the benefit extends beyond the assembled communication benchmark.

  4. Table 1 reports (2.4pm0.3%) Core SMR for d’Ascoli + LM at (k=1). For five evaluations on 100 sentences with mean 2.4%, the minimum possible sample standard deviation is approximately 0.55 percentage points. Could you provide the per-seed exact-match counts and clarify any additional averaging or use of standard errors?

Reviewer_so2COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, shortcut learning (timing) is considered in word-aligned and non-invasive brain-to-text (B2T) decoding. It is stated that jointly decoding words from overlapping and fixed-size MEG windows reveals timing information about words, making the B2T decoder capable of exploiting statistical regularities in word durations instead of purely using neural activity. The main experiment performed by the authors shows that jointly decoding words makes possible obtaining balanced word accuracies of 22.3% on real MEG data versus 22.0% on synthetic data with no brain signal but with overlap preserved, whereas lack of overlap reduces the accuracy on synthetic data to 5.8%. To resolve the problem, the paper suggests SimpleB2T which performs independent encoding of each word-aligned window of three seconds. The authors also combine it with aggregation over repeated observation and an autoregressive language-model prior. On a constraint clinically-motivated benchmark, SimpleB2T results in WER of 36.6% for five observations per word versus 74.3% of the neural-only baseline and 73.8% of the language-model-only baseline.

Critical strengths

S1: The paper presents a critical information leakage mechanism instead of just demonstrating higher performance. The synthetics control is well designed with model obtaining 22% balanced accuracy compared with 5.8% when the structure is removed. This clearly shows that the original joint-decoding mechanism allows a model to utilise/exploit non-neural timing information.

S2: The JS divergence and accuracy results are pointing in the same direction and are in sync. The components shuffled and noise control show the brain signal contributes. The ablation results are well defined. The limitations are well acknowledged regarding the lack of clinical validation and non-decoding imagined speech.

Critical weaknesses

W1: The paper directly cited improvements from the original d’Ascoli’s paper; without actually reproducing the numbers. Furthermore, the main claim regarding improvements is overgeneralised from one subject. Isolated MEG gets 9.5% against joint 22.3% (Fig. 1b). That gap shows the shortcut can explain a gain in this replication, not that the original reported gain was shortcut-driven. The original model may be trained differently here (reduced data, one subject, custom splits), and no evidence is given that the authors’ “Joint (MEG)” matches the published model.

W2: In Table 1, the authors compare a joint 200M model with an isolated model ~20M in size (page 5, lines ~266-267), using the stitched benchmark and a handwritten domain prompt. The paper does not do a side-by-side comparison with a jointly trained but equally sized model nor a very large isolated model, making it hard to say that time-based predictions were “less complementary to aggregating and LLM’s priors” (page 2, lines 099-107; page 8, lines 409-414).

W3: In Sec. 3.3 (pp. 6-7) and App. I (p. 20), sentences are LLM-written, and each position is populated with a word occurrence “irrespective of original sentence context.” Without any overlap, the joint decoder cannot take advantage of its shortcut, and Table 1 (p. 7) demonstrates d'Ascoli achieving ~98.9% WER for both k=1 and k=5, which is essentially a degenerate baseline. This demonstrates the failure of the joint model without its shortcut rather than the superiority of SimpleB2T on the original task, where cross-word context is valid. SimpleB2T is never tested on natural, overlapping, and actually-heard sentences. The k=5 condition (five occurrences of each word position), the known sentence length (p. 6) and the comparison to Moses et al. (2021) (p. 9), another task, another patient population, and another modality, further undermine the “approaching invasive performance” framing.

Questions for authors

Q1: Did you reproduce the baseline published number with your proposed training pipeline? Without this, the 22.3% vs 9.5% gaps performance improvement is not justified. Q2: The justification or quantification on how accurately the model can recover the onset to onset interval from overlapping window is needed specifically how the inter-word gaps are handling and the proportionate performance of explicit duration vs onset interval? Q3: The performance results reporting across independent subject and held out sessions are required to measure the generalized performance. Q4: Table 1 compares a 200M joint model with a 20M isolated model. Please include a ~20M joint model, and a ~200M isolated model, on the stitched benchmark and on natural sentences. Which factor drives the benefit on the clinical benchmark? Q5: Why is the LM combined with A_i(w) (Eq. 8) rather than log p_brain (Eq. 6, Fig. 2a)? How was T = 0.049 used in A_i? What is the performance with a naive-Bayes product of per-observation log probabilities, and with the standard calibrated log p_brain?

Reviewer_QJFVOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript investigates decoding of (perceived) speech from MEG recordings. The authors argue that a recent paper claiming high (for MEG) word-classification accuracy is in fact merely classifying words by the length of the corresponding MEG sequence (which leaked into the training data via the decoding setup). Blocking this "shortcut" reduces accuracy from 22% to under 10% (vs. a chance level of 2%). Indeed, a network trained in the same way on fake data achieves the same accuracy. But sentence decoding in a model trained with this shortcut blocked can be improved a great deal with a language model and by constructing "supertrials" (averaging responses across multiple stimulus presentations).

Critical strengths

That length information was being exploited by the original decoder was noted by d'Ascoli and colleagues, but the extent of this exploitation was not. Disseminating this fact to the community is critical. The authors' own decoder, although purely conventional, does achieve word error rates approaching usefulness. The paper is also written very clearly and is easy to follow. The major claims appear to be correct.

Critical weaknesses
  • The reporting of the manuscript's positive results is (to this reviewer's mind) somewhat misleading: The results show how to improve performance with prior knowledge and trial averaging, but not how to extract more speech information from MEG data. This raises some problems:

    • The use of the Qwen model to re-score sentences is somewhat artificial. The use of a large language model gives the impression of a very general prior over English---but then it turns out that the model was conditioned on a prompt explicitly specifying the domain of the sentences (Appendix L). Obviously this limits the scope of the results. But more unfortunately, it is not clear exactly how restrictive this is. E.g., for their LM, Moses et al. trained (I believe) an N-gram model on sentences composed from the 50-word vocabulary. The restriction is clear. But is this LM more or less restrictive than the Qwen setup in this paper? We can't tell. (But the separate reporting of performance on "core" and "expanded" sentences is certainly helpful in giving some intuition here.)

    • What is the balanced accuracy of the author's (no supertrials) decoder on the 92-word test data (for various k)? How does this scale with vocabulary size? (Perhaps this is in the supplement and I missed it?) This is the real test of information being extracted from MEG data. It is important to report because the LM re-scoring is not specific to MEG and could be applied to many different signals, like eye movements or EMG. So if our goal is to evaluate the usefulness of MEG for BCI, relative to other possible signals, we should compare bits extracted from MEG.

  • Along these lines: Is there a practical path from this setup to a BCI? There are several obstacles: the experiment decodes perceived, not produced, speech; the number of words per sentence is known (and used in the beam search); the word onset times are known; even on a 92-word vocabulary constrained to by a LM that knows the context (see prev.), WERs are well outside the useful range unless k=5 supertrials are used---and, on top of this, MEG machines fill a room. These problems interact: it is harder to get word onset times and to align trials for averaging under produced speech (presumably the ultimate target).

  • Similarly, what is the bit rate, and is it any better than (e.g.) the P300-speller?

  • The authors argue that experimental setups could facilitate identification of word onset---and, presumably, offset---times, the former being a requirement for their decoder. But if so, then what is wrong with using the duration of the response (to help) to decode words? This is not to say that the original d'Ascoli setup should be used. But can WERs be decreased further by, say, separately training a decoder based on word duration alone, and then mixing its outputs into the re-scoring (Eq. 9)? This would seem to make no more assumptions about what information is available than the current setup does.

MINOR

  • D'Ascoli and colleagues do note that "a linear regression trained to predict word length from the word embedding of the 5000 most common English words yields a Pearson correlation of 0.71." This should probably be acknowledged in this paper.
  • "100 observations per phoneme [were] used to improve decoding reliability, whereas SimpleB2T obtains large sentence-level gains from only five observations per word." These are not really comparable figures, since phonemes are much harder to identify than words (due to coarticulation and their brief duration--fewer samples at which to differ from one another).
Questions for authors

Please see the questions in the "Weaknesses" above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.