- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors show that a significant proportion of the performance reported in some prior non-invasive brain-to-text work can be attributed to a shortcut where the model uses the duration of the input rather than the brain activity in the input. This is shown relatively convincingly through controls comparing real brain activity to gaussian noise inputs of the same duration in a retrieval task. The authors also show that when fixed-duration windows are used, gaussian noise controls perform at chance level, whereas real brain activity does indeed show better than chance performance. I found the remaining content of the paper very hard to follow despite working within the same subfield.
Critical strengths
- The length duration shortcut is an interesting and useful finding
- In future speech BCI use, the sentence-level signal isn’t likely to be perfectly segmented, so duration shortcuts show that some papers may have led to misinterpreted progress in sentence-level neural decoding
- Figure 1 is generally very good and clearly lays out the problem
- It is excellent to see that using fixed-duration windows leads results with noise inputs to collapse effectively to chance level
Critical weaknesses
I think the paper should have focused on the strengths listed above, which are valuable insights. However, the rest of the content in the paper is highly confusing and difficult to follow (even most of the abstract). As a reviewer who works directly within the subfield of the paper, this is concerning and I encourage the authors to revise the scope and rewrite the paper from scratch accordingly. Unfortunately, the extent of rewriting required to make this work clear is too large of a change to be able to recommend acceptance, especially because rewriting is also a major prerequisite to being able to properly review the content of this work. I have otherwise tried to provide a best-effort review of the present content here.
The introduction begins by claiming non-invasive decoding has advanced rapidly by citing a line of work (Wang & Ji, 2022, Duan et al. 2023, etc.) that has been shown to suffer from significantly inflated results due to teacher-forced test results (Jo et al. 2026). I would strongly reconsider the citations used here as they do not reflect the claim.
The authors have shown the effect of the duration shortcut in retrieval, however, they have not shown whether this effect holds for downstream generated text in terms of WER/CER/BERTScore. Could the authors please show this? If gaussian inputs do not provide a good control here, then the authors could consider using permuted MEG inputs (as suggested in the “questions for authors” section).
The authors may want to consider citing closely related work on shortcuts in non-invasive M/EEG speech decoding. Specifically, in [A], the authors find a related but distinct shortcut where models can exploit the duration of words rather than sentences. I will not hold this against the authors as [A] is fairly recent and recommend it only as closely related work.
[A] Jayalath, Dulhan, and Oiwi Parker Jones. "Removing Timing Shortcuts Improves Non-Invasive Brain-to-Text." arXiv preprint arXiv:2609.40359 (2026).
While the authors' finding is relevant, the impact of the authors' work seems somewhat limited as it highlights a potential flaw in one ACL Findings paper and another arXiv preprint paper. The other papers highlighted have already been shown to have major flaws related to teacher-forced test results. Could the authors expand regarding the scope of the impact of the flaw they’ve found?
Questions for authors
Instead of using gaussian noise controls (or higher moments) with matched-duration, why not swap the corresponding MEG input for another of similar length? This effectively provides you with a permutation test that can determine if duration is the cause of inflated results, without using a different type of signal.
What is the purpose of the “lag” delta = 150ms in equation (3)?
8 comments
Response to Reviewer 2A42 [2/2]
W2 (citations at the start of the introduction)
Thank you for pointing this out; we have changed the citations. The first sentence now cites Tang et al. (2023), Défossez et al. (2023) and Lévy et al. (2026). The systems you mention are now cited in the second sentence, which is about the accuracy they report (L043–045).
W3 and Q1 (the shortcut in the generated text; permutation control)
Thank you for these two suggestions. We ran both controls on the generated text, Gaussian noise and another sentence's MEG of the same length, and added them to the paper (Sec. 4.2, L353–356; Appendix C.2).
The effect holds in the generated text: from another sentence's MEG of the same length, the pipeline generates the same text as from the real MEG, word for word, for 81% of sentences. Scored against the sentence the MEG came from, that text has a WER of 96.2. The text follows the length of the input, not the recording; the permutation control confirms that duration is the cause.
A is the test sentence, and B is the other one; all values are in %:
| Input | Scored against | R@1, before the decoder | WER | CER | BERTScore |
|---|---|---|---|---|---|
| A's own MEG | A | 58.4 | 30.8 | 22.0 | 94.7 |
| B's MEG, at A's length | A | 57.8 | 31.3 | 22.6 | 94.7 |
| B's MEG, at A's length | B | 7.6 | 96.2 | 74.7 | 85.9 |
| Gaussian noise, at A's length | A | 33.0 | 98.4 | 74.5 | 86.0 |
Gaussian noise of the same length keeps R@1 at 33.0% before the decoder (chance 0.15%), and its generated text is far from the target. The permutation control confirms the conclusion of the noise control when real MEG is used as input. The full table is in the General Response (Table R2).
W5 (scope of impact)
The shortcut comes from the low signal-to-noise ratio and the variable-length input; it does not depend on teacher forcing. Systems that no longer use teacher forcing have kept the same input: a sentence-level input, padded to a common length and mapped to text by one sequence-to-sequence model, carried over from paper to paper (e.g., Wang & Ji, 2022; Duan et al., 2023; Li et al., 2025; Liu et al., 2025; Yang et al., 2026b; Wang et al., 2026; the four most recent report their main results without teacher forcing). The risk therefore runs along a whole line of work (General Response R1). The work you recommend establishes its scope in the same way: it demonstrates the shortcut with one decoder on one dataset, and determines from the design of the other datasets which of them are affected.
A likely reason the shortcut went unnoticed is that padding to a common length is standard elsewhere in machine learning, for example in speech recognition, so its effect on a weak neural signal was not questioned (Related Work, L126–129). This is why the second half of the paper gives general controls and a criterion, which we hope will become a routine step toward trustworthy decoding.
Q2 (the lag of 150 ms in Eq. 3)
The lag is part of the preprocessing we take from Défossez et al. (2023), together with the window length and the sampling rate, and is cited with them after Eq. 3. Their reason for it: "Given the expected delay between a stimulus and its corresponding brain responses, we further shift the input brain signal by 150 ms into the future" (Défossez et al. (2023), Methods, "Residual dilated convolutions").
Summary of revisions. For W1, the abstract now states the steps of the paper. For W2, we changed the citations in the first two sentences of the introduction. For W3 and Q1, we added the permutation control and the metrics on the generated text (Appendix C.2, Table 4), and Sec. 4.2 reports them. For W4, Related Work now cites [A]. The structure and results of the paper are unchanged.
Thank you again for the time and effort you have given to reviewing our paper. Please let us know if any concerns remain, and we will be glad to explain further. If our responses resolve your concerns about the paper, we would be grateful if you would reconsider your rating.
Thank you for your response. I have raised my score to 2 in light of your response to W3 and W5. It is good to see validation of the shortcut on relevant downstream metrics. I also agree with the authors that sentence-level inputs are common and as such many works may suffer from the same shortcut. My concern about the clarity of the paper remains.
Response to Reviewer 2A42 [1/2]
Dear Reviewer 2A42,
Thank you for the review and for the comments on presentation. We are sorry to have put you through a frustrating read, and we explain the paper below (W1). We are grateful that you nevertheless recognised its core findings: the length shortcut as "an interesting and useful finding", and the collapse of noise to chance with fixed-duration windows. The additional results and explanations follow below.
W1 (the rest of the paper)
A sentence is a variable-length input, and in non-invasive decoding its length easily becomes a training shortcut: a decoder can match the two modalities based on signal length without learning the neural content. The first part of the paper finds this problem with controlled experiments and explains it.
Controlling the shortcut requires a fixed length, which also means that the model sees only a local window and cannot use information over a longer range. The rest of the paper therefore answers two questions: how well a local window can be decoded once the shortcut is controlled, and how to tell whether the gain from adding context comes from the local neural evidence or from a structural shortcut.
These are the same steps as in the work you recommend [A] (see W4): find the shortcut, decode each window independently to see what information is there, add context back, and check its gain. The last three are the second half of our paper. You noted that most of the abstract was hard to follow; it now states these steps, e.g., "Scoring windows alone gives up context, so we add it back as a separate stage on the window scores, where sentence length can return as a cue" (L027–028).
The introduction and Fig. 1 follow the same logic. Thank you for noting that Fig. 1 "clearly lays out the problem"; the whole paper is organised around the three sources it shows: structure in the input, evidence within a window, and context across windows.
We think the paper is more useful this way than if it stopped at finding the shortcut, explaining it, and reporting what remains once it is controlled. Its second half provides a set of control experiments for shortcuts of this kind: where in the model to look, what to change, and what result distinguishes a shortcut from a gain from the recording.
As you note, "some papers may have led to misinterpreted progress in sentence-level neural decoding". We agree, and the input design is shared well beyond the two papers mentioned in the review (W5). For this reason, we think a set of controls that any group can run on its own system is more useful than showing the problem paper by paper: they require no change to a pipeline's task, architecture or training.
W4 (Jayalath & Parker Jones, 2026)
Thank you for the recommendation; we had read this preprint when it appeared, and we now cite it in Related Work (L142–144). It follows the same logic and reports a shortcut that arises from joint decoding, together with a remedy.
We see a common origin for the shortcuts in the two papers, and the same remedy. In natural speech, context varies in length and in timing: sentences differ in duration, and words differ in the interval to the next word. When a model is trained on such context directly, this variation is a cue, and with the low signal-to-noise ratio of non-invasive recordings, it is the easier thing to learn. The remedy is the same in both papers: the model is given one fixed-duration window at a time and decodes it on its own, so neither the sentence length nor the interval between words is in its input. Context is brought in afterwards, when the window scores are combined.
That shortcut is therefore not present in our setting. As that paper states, it needs windows that are decoded jointly; each of ours is decoded alone (Sec. 3.3, Eq. 5). Our contextual stage has the form of their SimpleB2T with a different prior: they add the log-probability of a pretrained language model to the word scores, and we add a prior over the candidate sentences, pooled from the scores of the windows of the same sentence (Eq. 7).
Both papers find the shortcut with the same kind of control: the neural content is replaced while the input structure is kept, and the score is read before any language model. In that paper, a synthetic signal containing no brain activity, cut into windows at the true word onsets, reaches 22.0% with joint decoding, versus 22.3% on MEG. In ours, noise that keeps only the duration reaches 66.3%, compared with 90.8% on MEG (Sec. 4.2). The second half of our paper asks the same of our own contextual stage: its gain falls as neural evidence is removed and is zero without it (Sec. 4.4).