acceptodds
Under review as a conference paper at ICLR 2027

What Are We Actually Decoding? Source Attribution for Non-Invasive Brain-to-Language Retrieval

Abstract

Non-invasive neural language decoding has advanced rapidly, with end-to-end systems now turning MEG and EEG into fluent sentences. However, a high score may come from the decoder's language prior, the text metric, or the input structure rather than the brain, and the output alone cannot distinguish them. In this work, we ask where such scores come from. We find that in an end-to-end sentence decoding pipeline, input duration alone supports most of the score: scored before the decoder, Gaussian noise of the same duration keeps **66.3% sentence-retrieval R@1 against 90.8% for real MEG**, a shortcut absent from an earlier invasive ECoG control and missed by noise controls scored at the output. We propose **source attribution**, which reads the score out at the decoder's input, where evaluation becomes retrieval with a known chance level, and gives each of three sources—structure in the input, evidence within a window, and context across windows—its own control. With duration removed, word-locked MEG windows still identify their audio among 1,464 candidates for **44% of windows**, while noise stays at chance. We further state that a contextual gain can be credited to the brain only if it **falls with this local evidence and vanishes without it**. A training-free aggregation meets this criterion on two MEG datasets, raising R@1 by 7.7 and 6.9 points, and yields no spurious gain on EEG, where local evidence is very weak; a stage that uses sentence length in place of evidence fails it. Crucially, source attribution requires no change to a pipeline's task, architecture, or training, suggesting broad applicability across encoder–decoder pipelines. By turning whether a decoder reads the brain from an assumption into a measurement, it lays a foundation for trustworthy non-invasive language decoding. [Project Page](https://whatarewedecoding.github.io/) [Code](https://anonymous.4open.science/r/AuditNeuralDecoding/)

Primary area:
applications to neuroscience & cognitive science
Keywords:
neural language decoding, MEG, evaluation, source attribution, shortcut learning, brain-computer interfaces

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2A42OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors show that a significant proportion of the performance reported in some prior non-invasive brain-to-text work can be attributed to a shortcut where the model uses the duration of the input rather than the brain activity in the input. This is shown relatively convincingly through controls comparing real brain activity to gaussian noise inputs of the same duration in a retrieval task. The authors also show that when fixed-duration windows are used, gaussian noise controls perform at chance level, whereas real brain activity does indeed show better than chance performance. I found the remaining content of the paper very hard to follow despite working within the same subfield.

Critical strengths
  • The length duration shortcut is an interesting and useful finding
  • In future speech BCI use, the sentence-level signal isn’t likely to be perfectly segmented, so duration shortcuts show that some papers may have led to misinterpreted progress in sentence-level neural decoding
  • Figure 1 is generally very good and clearly lays out the problem
  • It is excellent to see that using fixed-duration windows leads results with noise inputs to collapse effectively to chance level
Critical weaknesses

I think the paper should have focused on the strengths listed above, which are valuable insights. However, the rest of the content in the paper is highly confusing and difficult to follow (even most of the abstract). As a reviewer who works directly within the subfield of the paper, this is concerning and I encourage the authors to revise the scope and rewrite the paper from scratch accordingly. Unfortunately, the extent of rewriting required to make this work clear is too large of a change to be able to recommend acceptance, especially because rewriting is also a major prerequisite to being able to properly review the content of this work. I have otherwise tried to provide a best-effort review of the present content here.

The introduction begins by claiming non-invasive decoding has advanced rapidly by citing a line of work (Wang & Ji, 2022, Duan et al. 2023, etc.) that has been shown to suffer from significantly inflated results due to teacher-forced test results (Jo et al. 2026). I would strongly reconsider the citations used here as they do not reflect the claim.

The authors have shown the effect of the duration shortcut in retrieval, however, they have not shown whether this effect holds for downstream generated text in terms of WER/CER/BERTScore. Could the authors please show this? If gaussian inputs do not provide a good control here, then the authors could consider using permuted MEG inputs (as suggested in the “questions for authors” section).

The authors may want to consider citing closely related work on shortcuts in non-invasive M/EEG speech decoding. Specifically, in [A], the authors find a related but distinct shortcut where models can exploit the duration of words rather than sentences. I will not hold this against the authors as [A] is fairly recent and recommend it only as closely related work.

[A] Jayalath, Dulhan, and Oiwi Parker Jones. "Removing Timing Shortcuts Improves Non-Invasive Brain-to-Text." arXiv preprint arXiv:2609.40359 (2026).

While the authors' finding is relevant, the impact of the authors' work seems somewhat limited as it highlights a potential flaw in one ACL Findings paper and another arXiv preprint paper. The other papers highlighted have already been shown to have major flaws related to teacher-forced test results. Could the authors expand regarding the scope of the impact of the flaw they’ve found?

Questions for authors

Instead of using gaussian noise controls (or higher moments) with matched-duration, why not swap the corresponding MEG input for another of similar length? This effectively provides you with a permutation test that can determine if duration is the cause of inflated results, without using a different type of signal.

What is the purpose of the “lag” delta = 150ms in equation (3)?

8 comments
AuthorsOpenReview ↗

Response to Reviewer 2A42 [1/2]

Dear Reviewer 2A42,

Thank you for the review and for the comments on presentation. We are sorry to have put you through a frustrating read, and we explain the paper below (W1). We are grateful that you nevertheless recognised its core findings: the length shortcut as "an interesting and useful finding", and the collapse of noise to chance with fixed-duration windows. The additional results and explanations follow below.

W1 (the rest of the paper)

A sentence is a variable-length input, and in non-invasive decoding its length easily becomes a training shortcut: a decoder can match the two modalities based on signal length without learning the neural content. The first part of the paper finds this problem with controlled experiments and explains it.

Controlling the shortcut requires a fixed length, which also means that the model sees only a local window and cannot use information over a longer range. The rest of the paper therefore answers two questions: how well a local window can be decoded once the shortcut is controlled, and how to tell whether the gain from adding context comes from the local neural evidence or from a structural shortcut.

These are the same steps as in the work you recommend [A] (see W4): find the shortcut, decode each window independently to see what information is there, add context back, and check its gain. The last three are the second half of our paper. You noted that most of the abstract was hard to follow; it now states these steps, e.g., "Scoring windows alone gives up context, so we add it back as a separate stage on the window scores, where sentence length can return as a cue" (L027–028).

The introduction and Fig. 1 follow the same logic. Thank you for noting that Fig. 1 "clearly lays out the problem"; the whole paper is organised around the three sources it shows: structure in the input, evidence within a window, and context across windows.

We think the paper is more useful this way than if it stopped at finding the shortcut, explaining it, and reporting what remains once it is controlled. Its second half provides a set of control experiments for shortcuts of this kind: where in the model to look, what to change, and what result distinguishes a shortcut from a gain from the recording.

As you note, "some papers may have led to misinterpreted progress in sentence-level neural decoding". We agree, and the input design is shared well beyond the two papers mentioned in the review (W5). For this reason, we think a set of controls that any group can run on its own system is more useful than showing the problem paper by paper: they require no change to a pipeline's task, architecture or training.

W4 (Jayalath & Parker Jones, 2026)

Thank you for the recommendation; we had read this preprint when it appeared, and we now cite it in Related Work (L142–144). It follows the same logic and reports a shortcut that arises from joint decoding, together with a remedy.

We see a common origin for the shortcuts in the two papers, and the same remedy. In natural speech, context varies in length and in timing: sentences differ in duration, and words differ in the interval to the next word. When a model is trained on such context directly, this variation is a cue, and with the low signal-to-noise ratio of non-invasive recordings, it is the easier thing to learn. The remedy is the same in both papers: the model is given one fixed-duration window at a time and decodes it on its own, so neither the sentence length nor the interval between words is in its input. Context is brought in afterwards, when the window scores are combined.

That shortcut is therefore not present in our setting. As that paper states, it needs windows that are decoded jointly; each of ours is decoded alone (Sec. 3.3, Eq. 5). Our contextual stage has the form of their SimpleB2T with a different prior: they add the log-probability of a pretrained language model to the word scores, and we add a prior over the candidate sentences, pooled from the scores of the windows of the same sentence (Eq. 7).

Both papers find the shortcut with the same kind of control: the neural content is replaced while the input structure is kept, and the score is read before any language model. In that paper, a synthetic signal containing no brain activity, cut into windows at the true word onsets, reaches 22.0% with joint decoding, versus 22.3% on MEG. In ours, noise that keeps only the duration reaches 66.3%, compared with 90.8% on MEG (Sec. 4.2). The second half of our paper asks the same of our own contextual stage: its gain falls as neural evidence is removed and is zero without it (Sec. 4.4).

AuthorsOpenReview ↗

Response to Reviewer 2A42 [2/2]

W2 (citations at the start of the introduction)

Thank you for pointing this out; we have changed the citations. The first sentence now cites Tang et al. (2023), Défossez et al. (2023) and Lévy et al. (2026). The systems you mention are now cited in the second sentence, which is about the accuracy they report (L043–045).

W3 and Q1 (the shortcut in the generated text; permutation control)

Thank you for these two suggestions. We ran both controls on the generated text, Gaussian noise and another sentence's MEG of the same length, and added them to the paper (Sec. 4.2, L353–356; Appendix C.2).

The effect holds in the generated text: from another sentence's MEG of the same length, the pipeline generates the same text as from the real MEG, word for word, for 81% of sentences. Scored against the sentence the MEG came from, that text has a WER of 96.2. The text follows the length of the input, not the recording; the permutation control confirms that duration is the cause.

A is the test sentence, and B is the other one; all values are in %:

InputScored againstR@1, before the decoderWERCERBERTScore
A's own MEGA58.430.822.094.7
B's MEG, at A's lengthA57.831.322.694.7
B's MEG, at A's lengthB7.696.274.785.9
Gaussian noise, at A's lengthA33.098.474.586.0

Gaussian noise of the same length keeps R@1 at 33.0% before the decoder (chance 0.15%), and its generated text is far from the target. The permutation control confirms the conclusion of the noise control when real MEG is used as input. The full table is in the General Response (Table R2).

W5 (scope of impact)

The shortcut comes from the low signal-to-noise ratio and the variable-length input; it does not depend on teacher forcing. Systems that no longer use teacher forcing have kept the same input: a sentence-level input, padded to a common length and mapped to text by one sequence-to-sequence model, carried over from paper to paper (e.g., Wang & Ji, 2022; Duan et al., 2023; Li et al., 2025; Liu et al., 2025; Yang et al., 2026b; Wang et al., 2026; the four most recent report their main results without teacher forcing). The risk therefore runs along a whole line of work (General Response R1). The work you recommend establishes its scope in the same way: it demonstrates the shortcut with one decoder on one dataset, and determines from the design of the other datasets which of them are affected.

A likely reason the shortcut went unnoticed is that padding to a common length is standard elsewhere in machine learning, for example in speech recognition, so its effect on a weak neural signal was not questioned (Related Work, L126–129). This is why the second half of the paper gives general controls and a criterion, which we hope will become a routine step toward trustworthy decoding.

Q2 (the lag of 150 ms in Eq. 3)

The lag is part of the preprocessing we take from Défossez et al. (2023), together with the window length and the sampling rate, and is cited with them after Eq. 3. Their reason for it: "Given the expected delay between a stimulus and its corresponding brain responses, we further shift the input brain signal by 150 ms into the future" (Défossez et al. (2023), Methods, "Residual dilated convolutions").

Summary of revisions. For W1, the abstract now states the steps of the paper. For W2, we changed the citations in the first two sentences of the introduction. For W3 and Q1, we added the permutation control and the metrics on the generated text (Appendix C.2, Table 4), and Sec. 4.2 reports them. For W4, Related Work now cites [A]. The structure and results of the paper are unchanged.

Thank you again for the time and effort you have given to reviewing our paper. Please let us know if any concerns remain, and we will be glad to explain further. If our responses resolve your concerns about the paper, we would be grateful if you would reconsider your rating.

Thank you for your response. I have raised my score to 2 in light of your response to W3 and W5. It is good to see validation of the shortcut on relevant downstream metrics. I also agree with the authors that sentence-level inputs are common and as such many works may suffer from the same shortcut. My concern about the clarity of the paper remains.

Reviewer_cZzxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines whether non-invasive neural language-decoding scores reflect neural evidence or structural shortcuts. It introduces a retrieval-based evaluation framework covering input structure, within-window evidence, and cross-window aggregation. Duration-preserving noise retains 66.3% sentence-retrieval accuracy compared with 90.8% for real MEG in a reproduced pipeline. Separate fixed-window experiments demonstrate above-chance audio retrieval, while sentence-bucket aggregation improves R@1 by 7.7 and 6.9 percentage points on two MEG datasets. Noise, grouping perturbations, and graded evidence attenuation are used to examine what supports these gains.

Critical strengths

The duration-preserving noise experiment identifies a consequential evaluation shortcut. Retaining 66.3% sentence-retrieval accuracy after replacing neural values, while preserving padding and masks, demonstrates that high accuracy can arise without stimulus-specific neural content. This provides a concrete reason to reconsider how noise controls are constructed and interpreted.

The aggregation analysis includes complementary controls, three-seed MEG results, and a second MEG dataset evaluated with the same aggregation configuration. Holding local logits fixed makes the comparisons interpretable, while the attenuation experiments provide useful evidence that aggregation gains depend on stimulus-specific information.

Critical weaknesses

The headline pipeline diagnostic does not fully isolate the effects attributed to duration and readout location. Figure 2a and Table 3 change both input duration and prediction task, while Section 4.2 contrasts earlier tests that differ in both noise construction and evaluation location. Appendix C.1 also leaves the sentence-retrieval scoring procedure unspecified. Together, these issues limit reproducibility and weaken the broader claim that reading before the decoder is what exposes the shortcut.

The framework establishes signal dependence more clearly than source separation. A contextual gain that disappears under noise may still depend on interactions between neural evidence and identifying sentence structure; a length-based prior gated by neural confidence could exhibit this behaviour. Consequently, the intervention-induced changes in R@1 do not uniquely determine what the score owes to each source, as the interpretation in Section 3.2 suggests.

Questions for authors
  1. Does the unnormalised aggregation variant in Table 8, which reaches 71.2% R@1, also pass the evidence-attenuation criterion? Applying the same controls to this variant would help clarify what the criterion excludes beyond a purely signal-independent shortcut, particularly given the structural dependence motivating normalisation.

  2. Can the duration diagnostic keep the sentence candidates and scoring procedure fixed while removing the duration cue? The current comparison changes from sentence retrieval to prediction over 2,048 codes, and the fixed-duration noise score remains 1.53% against 0.05% uniform chance. A code-frequency baseline would also help interpret this residual accuracy without assuming that it demonstrates leakage.

  3. Can both noise interventions—replacing the full padded tensor and replacing only values within the valid duration—be evaluated at both the representation readout and generated-text output on the same checkpoint? This would distinguish whether earlier tests missed the shortcut because they evaluated after the decoder or because their intervention removed the duration cue.

  4. How are sentence candidates represented and scored in Appendix C.1, including temporal pooling and treatment of padded time steps? The description of quantising Whisper encoder representations also needs reconciling with BrainECHO’s spectrogram-autoencoder codebook, which precedes Whisper in the original architecture. Please specify the exact measurement point and any architectural or training changes.

  5. Were overlapping neural and audio intervals excluded across every split pair for MOUS and Brennan, and between training and validation for Gwilliams? Appendix A.1 explicitly describes overlap removal only with respect to Gwilliams test windows. Distinct stimulus identities alone do not guarantee disjoint samples when three-second, word-aligned windows extend across sentence boundaries.

  6. How do the proposed controls extend SemKey’s evaluation, which already combines retrieval-based evaluation of generated text with graded corruption of EEG embeddings? The decoder-input readout and explicit grouping controls differ from those procedures, but Section 2 currently cites this work only in the padding discussion. Explaining which additional confounds these controls expose would clarify the methodological contribution.

  7. How does the approximately chance-level proportion of same-sentence top-1 errors in Section 4.3 establish that local representations lack sentence context? Context could improve the correct window’s rank without making incorrect predictions favour other windows from its sentence, particularly when each input already spans three seconds. The conclusion appears better restricted to an absence of same-sentence error bias.

5 comments
AuthorsOpenReview ↗

Response to Reviewer cZzx [1/2]

Dear Reviewer cZzx,

Thank you for the careful and constructive review, and for recognising that the result gives "a concrete reason to reconsider how noise controls are constructed and interpreted". Several of the new results in the paper come directly from your questions.

W1, with Q2, Q3 and Q4 (isolating duration and readout; scoring)

Thank you for raising this. We isolated each factor as you suggested, and the result holds under every control. The full tables are in the General Response (Tables R1 and R2).

Q2 (removing duration with the candidates and scoring fixed)

Yes. With the scoring kept and the duration fixed, noise is at chance (0.04% R@1; chance 1/1,665 = 0.06%), and real MEG reaches 0.59% (General Response R2; Appendix C.2).

The code-frequency baseline is 1.53%, the same as the noise score: under noise, the prediction falls on the most frequent code (the uniform level is 1/2048). The residual accuracy is therefore not leakage but that of an input-blind model, as Sec. 4.2 states (L323): "With fixed-duration inputs, noise no longer scores high." The retrieval result above shows the same with a chance level that does not depend on code frequency. Thank you for suggesting this baseline; it gives the conclusion a second, independent line of support.

The result also shows what this pipeline reads once duration is removed. Real MEG is about ten times chance, so neural information is present, but far below the level that would support decoding. With the preprocessing and local model of Sec. 4.3, fixed-duration windows reach 44% among 1,464 candidates.

Q3 (noise construction and readout location)

With noise as the control, either one alone is enough to hide the shortcut. The four cells you propose separate the two explanations cleanly, and we have added them to the paper (Appendix C.2). They come from the same generative pipeline, and the full table is in the General Response (Table R2).

A test that replaces the whole padded tensor misses the shortcut wherever the score is read: it is at chance before the decoder (0.2%) and far from the target at the output. A test scored on the generated text misses it whichever noise is used: WER is 98.4 with noise within the valid length and 112.5 with noise over the whole tensor.

Earlier tests did both (Sec. 4.2, L351), and Sec. 4.2 now states each condition by itself (L353–355).

Q4 (candidates, scoring, and measurement point)

Scoring uses no temporal pooling and does not count padded time steps. Each candidate sentence is represented by its code sequence and ranked by code accuracy against the predicted sequence, over the candidate's valid columns. Appendices C.1 and C.2 share this scoring, and Appendix C.2 of the revision states it.

The conclusion is the same at both measurement points. In C.1, we cut the pipeline after the Whisper encoder and quantise its output; the only change is the quantiser, trained jointly with the pipeline. Following your comment, we also read the codes at the mel spectrogram, with the pipeline unchanged (Appendix C.2): 58.4% R@1 for real MEG and 33.0% for noise within the valid length.

W1, last point (reading before the decoder)

Reading before the decoder makes the check faster and more reliable, which is a key advantage for a control experiment.

Before the decoder, one metric and plain noise are enough. Real MEG, noise of the same length, and chance are at 58.4%, 33.0% and 0.15%. Noise far above chance shows the shortcut directly. In addition, the decoder does not have to be run, and generating text is the slow and costly part of the pipeline.

At the output, noise does not show the shortcut, and several metrics have to be read together. The same noise gives a WER of 98.4, and real MEG gives 30.8. A text metric has no chance level, so these numbers suggest that the model needs the brain signal. Earlier tests missed the shortcut for this reason. The permutation control shows it: from another sentence's MEG of the same length, the pipeline generates the same text as from the real MEG, word for word, for 81% of sentences. That text has to be scored against both sentences: its WER is 31.3 against the target and 96.2 against the sentence the MEG came from. The metric also matters: a fixed output such as "the the" already reaches a BERT-F1 of 0.80–0.87 (General Response, Table R2).

W2 (what the controls measure)

Thank you for this distinction. This is how we use the controls. We have modified the first sentence of Sec. 3.2 to avoid the misunderstanding: "the drop in score shows how much it depends on that source" (L188–189).

AuthorsOpenReview ↗

Response to Reviewer cZzx [2/2]

Q1 (aggregation without normalisation)

Without normalisation, the gain goes to 0 (from +26.6) without evidence, and this is not a length shortcut.

Long sentences gain more because they have more windows, and so more evidence, not because their length is recognised: with the same number of query windows for every sentence (8), long and short sentences gain almost the same (Appendix D.3, Table 12).

Sentence length (windows)8–1212–1616–2121–43
Gain without normalisation, 8 windows each (points)23.824.824.725.2

We also constructed the case of "interactions between neural evidence and identifying sentence structure" in two forms: a length-based prior that selects the candidates before the neural evidence ranks them (Appendix E.5), and the same prior gated by neural confidence. The criterion detects both:

Gain in R@1 (points)Clean MEGHalf the evidenceNo evidence
Aggregation+7.7+2.30.0
Aggregation without normalisation+26.6+18.30.0
Length-based prior+5.6+16.8+1.5
Length-based prior, gated by neural confidence−3.8−0.1+0.1

Without a gate, the prior's gain does not vanish without the evidence (+1.5, against 0.0 for the aggregation). A length-based prior still narrows the candidates to the sentences of matching length: its prior over sentences is a function of the length, P(sentence | length); the aggregation's is a function of the evidence, P(sentence | evidence).

With a gate, the prior's gain does not fall with the evidence (−3.8, −0.1, +0.1). A gate can close the prior where there is no evidence; it does not make a prior built on length follow the evidence. A stage that takes a shortcut ignores the content (in the pipeline of Sec. 4.2, another sentence's MEG gives the same text for 81% of sentences). The criterion therefore identifies this case as well. Thank you for raising it; it strengthens the result.

Q5 (overlap across split pairs)

No test window overlaps a training or validation window on Gwilliams or MOUS, in neural or in audio time. On Brennan, 145 of the 12,600 test windows (1.15%) overlap in EEG time only, and excluding them leaves the results unchanged. Appendix A.1 of the revision states this.

Between training and validation, windows at sentence boundaries can overlap, but the validation set is used only to select checkpoints and hyperparameters, and every reported result is computed on the test windows. To keep the training and validation sentences intact and use as much of the data as possible, overlap was removed with respect to the test set only.

Q6 (relation to SemKey's evaluation)

Two confounds: the input's duration, which these controls expose, and the decoder, which they keep out of the score.

Input duration. A score can depend on the input through its length alone: with only the values replaced, the pipeline we reproduce keeps 33.0% R@1, against 0.2% when the structure is replaced as well. SemKey pads the sentence-level EEG to a common length in the same way, tying its input to the text length, and further predicts sentence length from the EEG as an explicit attribute given to the language model. Whether the duration shortcut we report is present there remains to be tested.

The decoder. SemKey's retrieval and noise tests are scored on the generated text, after the language model. Our readout is placed "at the decoder input, before the decoder's prior enters the score" (Sec. 3.1, L153), so neither the language model nor a text metric is part of it. The duration cue enters before any language model and is present whichever decoder follows; scored at the output, the same noise gives a WER of 98.4, and the cue is not seen (Q3).

At the contextual stage, the same cue appears as sentence length, and the grouping controls expose it (Sec. 4.4). Sec. 2 now discusses SemKey together with the other checks (L133–136).

Q7 (same-sentence errors in Sec. 4.3)

We agree, and Sec. 4.3 now states the conclusion in this restricted form: "Local errors thus show no bias toward the target's sentence" (L400). The controls in Sec. 4.4 test where the sentence-level gain comes from.

Summary of revisions. For W1 and Q2 to Q4, we added Appendix C.2 (Tables 4 and 5): the fixed-window test, the four cells, and the readout at the mel spectrogram with its scoring. Sec. 4.2 now states each condition by itself. For W2, we revised the first sentence of Sec. 3.2. For Q1, we added Tables 11 and 12 (Appendix D.3) and the gated prior (Appendix E.5). For Q5, Q6 and Q7, we revised Appendix A.1, Sec. 2 and Sec. 4.3.

Thank you again for your careful reading of the paper's details. Your questions have improved it and helped validate its conclusions. If your concerns are resolved, we would be grateful if you would consider raising your rating.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer cZzx,

Thank you again for your constructive review. We have responded directly to each of your concerns and revised the manuscript accordingly. Before the discussion closes, we would appreciate a brief note on whether our responses address your concerns or if there is anything we can clarify further.

Best,
The authors

Reviewer_LMDxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether non-invasive neural language decoding scores reflect stimulus-specific neural evidence or exploitable input structure, claiming three contributions:

  • Identification of a substantial input-duration shortcut in a sentence-level decoding pipeline
  • A pre-decoder retrieval framework for attributing performance to input structure, local evidence, and cross-window context
  • An evidence-dependence criterion for assessing contextual gains, demonstrated through training-free sentence-bucket aggregation

Experiments provides a useful, empirically supported diagnostic framework for the tested settings, but its closed-set audio retrieval and externally supplied sentence grouping do not establish semantic decoding or validate the claimed general applicability across generation pipelines.

Critical strengths
  • The paper’s central intellectual contribution is to recognize that the source of decoding success is itself a scientific question, rather than something established by predictive accuracy. This perspective challenges an underlying assumption in interpreting decoding results and redirects attention toward what those results actually demonstrate, giving the work value beyond an incremental performance improvement.
  • The paper builds an explanatory argument in which its critique and proposed evaluation approach reinforce one another. This coherence makes the contribution cumulative: the experiments jointly support a way of reasoning about decoding claims, rather than a collection of isolated observations, strengthening the case for acceptance as a methodological contribution.
Critical weaknesses
  1. Weakness: The empirical scope of the shortcut analysis is narrower than the broader critique of neural language decoding that it is used to motivate.
    Evidence: In the reproduced subject-split pipeline, duration-preserving noise achieves 66.3% R@1 compared with 90.8% for real MEG, with evaluation sentences also appearing during training. This result demonstrates a substantial vulnerability in that specific protocol, but does not establish how prevalent or consequential the same issue is under unseen-stimulus evaluation or in generation pipelines.
    Why it matters: This gap limits the reach of the paper’s main methodological conclusion: a compelling diagnosis of one evaluation protocol does not by itself support a general reassessment of neural language decoding. If that broader claim is central to the submission’s significance, supporting it requires substantive additional validation; otherwise, the contribution should be assessed as a narrower protocol-specific finding.
Questions for authors

Clarification questions:

  • Is the duration shortcut intended as a finding specific to the reproduced subject-split protocol, or as an explanation for inflated performance more broadly in neural language decoding? Given that evaluation sentences also appear during training in this setting, what existing evidence separates the contribution of duration from its interaction with stimulus overlap?

Suggestions: None

5 comments
AuthorsOpenReview ↗

Response to Reviewer LMDx [1/2]

Dear Reviewer LMDx,

Thank you for the thoughtful review, and for recognising that where a decoding score comes from is itself a scientific question. The full results are in the General Response (Tables R1 and R2).

W1 (scope of the shortcut analysis)

The vulnerability you describe belongs to the input design and to the tests used with it, and both are carried over across many systems (e.g., Wang & Ji, 2022; Duan et al., 2023; Li et al., 2025; Liu et al., 2025; Yang et al., 2026b; Wang et al., 2026). The protocol decides only how strongly it shows in the test score (Q1).

The input is carried over. A sentence-level input padded to a common length carries the sentence length. Many of these works improve other components, such as the encoder, the language model, or the training objective. However, the structure that carries the risk of shortcut learning remains in place, so the risk persists across many published works.

The usual tests cannot show the shortcut. The noise tests reported for such systems replace the whole input, padding included, so the length is removed along with the signal. Other tests score the generated text, where the decoder's prior and embedding-based text metrics come in as further sources of information: a fixed output such as "the the" already reaches a BERT-F1 of 0.80–0.87 (Table 1b; Sec. 4.2, L351–359).

Two things generalise: the mechanism and the checks. You describe the contribution as "a way of reasoning about decoding claims, rather than a collection of isolated observations". This is how we see its scope, too. The pipeline we reproduce is the demonstration, and two things carry over from it.

  1. The mechanism. The paper explains why the shortcut arises (Sec. 4.2, L344–346), which tells us which systems are exposed: those with the same input structure. Some recently published results show the same pattern: for example, Yang et al. (2026b) report a BLEU-1 of 53.16 with sessions held out, but only 6.91 and 6.54 with stories and sentences held out (their Table 7). We therefore think that our analysis of the mechanism and the results we report are a useful reference for other work with this input.

  2. The checks. They need no change to the model or its training (abstract, L033–034), so any group can run them on its own system. Whether a given system takes the shortcut, and how much it matters, is what they measure.

A related shortcut was established the same way: Jayalath & Parker Jones (2026) demonstrate it with one decoder on one dataset, and determine, from the design of the other datasets, which are affected.

We ran both settings you named on the same pipeline.

Unseen sentences. Real MEG scores as noise does, near chance: what the pipeline learned in training is the length. The evidence is under Q1.

Generation. The MEG of another sentence of the same length produces the same sentences as the real MEG. The generated text is therefore chosen by the input length, and a good text score is reached without the content of the recording (General Response, Table R2).

For this question, the readout before the decoder and the generated text give the same answer, so the first can stand in for the second.

AuthorsOpenReview ↗

Response to Reviewer LMDx [2/2]

Q1 (duration and stimulus overlap)

Thank you for this question.

Stimulus overlap is standard practice and is not the problem. The problem is that the two modalities match in length. A sentence's recording is cut at the sentence's boundaries, so the neural input is as long as its target, and a model can pair the two by length without reading the neural content: ranking the candidates by length alone, with no model, already identifies 23.1% of the 661 sentences (chance 0.15%; General Response, Table R1). Under the subject split, the test recordings come from participants the model has never seen, who hear the same sentences as the training participants. This is the standard protocol, and it is not what goes wrong here.

The overlap lets the shortcut show in the reported score. When the test sentences were heard in training, each length points to a sentence the model has stored. With whole stories held out, the lengths still match but point to nothing it has learned, and real MEG falls from 90.8% to 7.1% (Sec. 4.2, L345–346). The new results agree: with sentences held out, real MEG scores as noise does (2.0% against 1.2%). The length is still in the input, since ranking by it alone identifies 56.9% of these sentences; what the model learned is which training sentence goes with which length.

The overlap does not create the shortcut. In training, the model takes it under every split. On its own training sentences, another sentence's MEG of the same length scores as the real MEG does: 57.7%, 11.9% and 7.8% under the subject, stimulus and story splits, against 57.8%, 11.3% and 8.1% (chance at most 0.22%; Appendix C.2). Holding sentences out therefore takes the shortcut out of the test score and leaves it in the model.

The shortcut goes unnoticed because a low score on unseen sentences is usually read as difficulty: the pipeline we reproduce (Li et al., 2025) notes that "testing with completely unseen sentences can be overly ambitious". The input design then stays in use in recent systems (Liu et al., 2025; Wang et al., 2026). Yet the neural content can be decoded once the cue is removed: on the same dataset, fixed-duration windows with held-out stimuli rank first among 1,464 candidates for 44.4% of windows.

Remark in the summary (retrieval readout, sentence grouping, generation pipelines)

We have added the results on generated text, and the shortcut is there as well (General Response, Table R2; Appendix C.2). They also show how much it takes to see the problem at the output: a table of several text metrics, and a series of comparisons that each change one thing. With sentences held out, a fifth to nearly a third of the generated texts are word-for-word training sentences (19.7% and 29.1%), and noise of the same length gives as many, so the output alone does not show where a score comes from.

Before the decoder, the same pipeline gives a much simpler picture. The test targets are fixed and finite, so a representation can only be ranked against them. Retrieval is not a task we chose in place of generation; it is what a generation pipeline offers at this point: "Retrieval-based and generation-based decoding meet here: the former stops, and the latter continues" (Sec. 3.1, L160–161). There, three numbers are enough: real MEG, noise of the same length, and chance, at 58.4%, 33.0% and 0.15%, from the same run as the text results. Every encoder–decoder pipeline has this point, so the same reading can be taken in any of them. Reading the score there takes nothing away from generation, and the pipeline can still generate afterwards.

Yes, the sentence grouping is supplied from outside, and the criterion does not depend on it. We use it to build an example on which the criterion can be tested (Sec. 3.4, L253); the criterion is stated for any contextual stage, wherever its grouping comes from; the aggregation is there to show that a stage can meet it.

We agree: the paper does not establish semantic decoding. Our concern comes before it: can the decoder read the neural content at all, whether that content is semantic or phonetic? This is why the result matters. A pipeline that gives good output without reading the neural content cannot be taken as evidence of better neural decoding, at any level of language.

Summary of revisions. For W1 and Q1, we added Appendix C.2 (Tables 4 and 5): the pipeline is retrained under three splits and read out on its training sentences, and its generated text is scored with the permutation control. Sec. 4.2 now reports these results. For your remark on the retrieval readout, the Discussion now states why the score is read before the decoder.

We hope these additions close the gap you identified between the diagnosis and its scope, and we thank you again for helping us strengthen the paper. If your concerns are resolved, we would be grateful if you would consider raising your rating.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer LMDx,

Thank you again for your constructive review. We have responded directly to each of your concerns and revised the manuscript accordingly. Before the discussion closes, we would appreciate a brief note on whether our responses address your concerns or if there is anything we can clarify further.

Best,
The authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to All Reviewers [1/2]

We thank all reviewers for their thoughtful and constructive feedback. We are encouraged that they find the question important, the length shortcut a consequential finding, and the controls informative. The experiments they requested are now in the paper, and the result holds in each. Below, we first address two recurring points and then respond to reviewer-specific comments. Changes in the revised PDF are marked in blue, and the line numbers we cite refer to it; the works we cite are in its reference list.

R1. Scope of the length shortcut (LMDx: W1, Q1; 2A42: W5)

The shortcut comes from how the input is built, and this input is used across sentence-level decoders. It is a broad risk of shortcut learning, and the usual output tests do not show it (R2).

The input is shared. A sentence-level input padded to a common length carries the length of the sentence (Sec. 4.2, L344–345), and this design is carried over from paper to paper (e.g., Wang & Ji, 2022; Duan et al., 2023; Li et al., 2025; Liu et al., 2025; Yang et al., 2026b; Wang et al., 2026). The four most recent of these papers report their main results without teacher forcing, so correcting teacher forcing does not remove it.

The shortcut is learned under every split. We retrained the pipeline under three splits (new Appendix C.2).

Table R1 (Tables 4 and 5 of the revised paper). R@1 (%) before the decoder under three splits.

Split (candidates)Real MEGNoise within valid lengthOther sentence's MEG, same lengthLength only, no model
Subject (661)58.433.057.823.1
Stimulus, sentences held out (133)2.01.21.856.9
Story held out (120)1.61.91.759.8

With sentences heard in training, another sentence's MEG of the same length scores as the real MEG does (57.8% against 58.4%). With sentences held out, real MEG scores as noise does (2.0% against 1.2%, and 1.6% against 1.9%), although the length is still in the input (last column: ranking by length alone identifies 56.9% and 59.8%): what the model has learned is which training sentence goes with which length. Once length is removed from the input, the same dataset is decoded: fixed-duration windows with held-out stimuli rank first among 1,464 candidates for 44.4% of windows.

Two things carry over to other systems.

  1. The mechanism: the paper explains why the shortcut arises, which tells us which systems are exposed, namely those with this input.

  2. The checks: they need no change to a pipeline's task, architecture, or training (abstract, L033–034), so any group can run them on its own system, and whether a given system is affected is what they decide.

AuthorsOpenReview ↗

General Response to All Reviewers [2/2]

R2. Generated text, noise construction, and readout (2A42: W3, Q1; cZzx: W1, Q2, Q3)

The shortcut carries through to the generated text, and the usual output tests do not show it. Before the decoder, it appears with any replacement that preserves the input's structure.

Sec. 4.2 (L351–353) states that earlier noise tests "replace the whole input and score the output text", and that "with only the values within the valid length replaced and the score read before the decoder" the pipeline keeps 66.3%. The new results separate these conditions and add the generated text (new Appendix C.2). Quantisation is at the mel spectrogram, as in the original pipeline (Li et al., 2025); subject split: 661 candidate sentences (chance R@1 0.15%); text is generated with beam search. These values differ from the 90.8% and 66.3% of Sec. 4.2, which are quantised after the Whisper encoder with fewer candidates (chance 0.8%). Both show the same result: the shortcut is seen before the decoder.

Table R2 (Table 4 of the revised paper). All values are in %.

Test inputR@1WERCERBLEU-1BERTScore
Real MEG58.430.822.073.894.7
Other sentence's MEG, cropped to the target's length57.831.322.673.894.7
Same, scored against the other sentence7.696.274.721.585.9
Noise within valid length33.098.474.520.686.0
Noise over the whole padded tensor0.2112.591.13.683.4

The shortcut holds on the generated text (2A42: W3, Q1). Another sentence's MEG of the same length produces the same text as the real MEG: WER 31.3 vs 30.8, CER 22.6 vs 22.0, BERTScore 94.7 vs 94.7, and 81% of the generated sentences are identical. Scored against the sentence the MEG came from, it is close to the noise rows.

Noise tests at the output do not show it (cZzx: Q3). Both noise rows give text far from the target (WER 98.4 and 112.5), regardless of how the noise is built. The metric matters as well: BERTScore is still 83.4 for noise over the whole input, against 94.7 for the correct text, and fixed outputs such as "the the" already reach 0.80–0.87 (Table 1b; Sec. 4.2, L358–359).

Before the decoder, it is seen with either replacement that keeps the structure (cZzx: Q3). Noise within the valid length keeps 33.0% R@1, and another sentence's MEG keeps 57.8%, compared with 58.4% for the real MEG. Noise over the whole padded tensor removes the structure and falls to chance. Here "each target is a unique stimulus with a known chance level" (Sec. 4.2, L356–357), and neither the decoder nor the text metric enters the score.

With duration removed, the same scoring puts noise at chance (cZzx: Q2). With fixed 3.04 s windows ranked among 1,665 candidate windows, real MEG stays above chance and noise does not: R@1 is 0.59% for real MEG and 0.04% for noise (chance 0.06%), and R@10 is 4.40% and 0.59% (chance 0.60%).

AuthorsOpenReview ↗

Summary of the Discussion

Dear all reviewers and AC:


We sincerely thank all three reviewers for their careful evaluation and constructive feedback. We also thank the AC for the evaluation of this paper. As the discussion period draws to a close, we summarise the strengths recognised by the reviewers and our responses to their concerns.

Strengths recognised by the reviewers

  • LMDx described the work as a way of reasoning about decoding claims rather than a collection of isolated observations, and after the rebuttal continued to see the diagnostic framework as a genuine contribution, maintaining the rating of weak acceptance.
  • cZzx stated that the additional experiments substantially strengthen the empirical evidence and alleviate several concerns. After the rebuttal, cZzx raised the rating to weak acceptance.
  • 2A42 recognised the duration shortcut as an interesting and useful finding, and noted that sentence-level inputs are common and many works may suffer from the same shortcut. After the rebuttal, 2A42 raised the rating from 1 to 2.

Core concerns and responses

ConcernReviewerStatus and response
Scope of the shortcut; split and stimulus overlapLMDx, 2A42Addressed. We retrained the pipeline under three splits. The shortcut is learned under every split (Appendix C.2, Tables 4 and 5).
Shortcut in the generated text; permutation control2A42, LMDxAddressed. Another sentence's MEG of the same length gives the same generated text as the real MEG for 81% of sentences (Appendix C.2, Table 4).
Noise construction and readout locationcZzxAddressed. We added the four cells the reviewer proposed. Before the decoder, the shortcut appears with either replacement that keeps the input's structure (Appendix C.2, Table 4).
Fixed-duration comparisoncZzxAddressed. With the duration fixed, noise falls to chance, both among 1,665 candidate windows (0.04% R@1; chance 0.06%) and, as requested in the follow-up, under the conditions of Table 4 (0.1%; chance 0.15%) (Appendix C.2).
Scoring and measurement pointcZzxAddressed. Read at the mel spectrogram with the pipeline unchanged, the conclusion is the same (Appendix C.2).
What the contextual criterion measurescZzxAddressed. We added the aggregation without normalization and a gated length prior. The criterion tells the two cases apart (Appendix D.3, Tables 11 and 12).
Relation to Jayalath & Parker Jones (2026)2A42Addressed. Related Work now discusses the common origin of the two shortcuts.
Clarity of the writing2A42Revised. We revised the sentences the reviewer listed and similar sentences throughout the paper (marked in blue). The reviewer maintained this concern, and we will continue to improve the writing.

We have responded to all concerns raised by the reviewers, and no experimental concern remains open. The one remaining concern, raised by Reviewer 2A42, is the clarity of the writing, which we will continue to improve.


Sincerely,
Authors of Submission 3668

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.