What Are We Actually Decoding? Source Attribution for Non-Invasive Brain-to-Language Retrieval

Non-invasive neural language decoding has advanced rapidly, with end-to-end systems now turning MEG and EEG into fluent sentences. However, a high score may come from the decoder's language prior, the text metric, or the input structure rather than the brain, and the output alone cannot distinguish them. In this work, we ask where such scores come from. We find that in an end-to-end sentence decoding pipeline, input duration alone supports most of the score: scored before the decoder, Gaussian noise of the same duration keeps **66.3% sentence-retrieval R@1 against 90.8% for real MEG**, a shortcut absent from an earlier invasive ECoG control and missed by noise controls scored at the output. We propose **source attribution**, which reads the score out at the decoder's input, where evaluation becomes retrieval with a known chance level, and gives each of three sources—structure in the input, evidence within a window, and context across windows—its own control. With duration removed, word-locked MEG windows still identify their audio among 1,464 candidates for **44% of windows**, while noise stays at chance. We further state that a contextual gain can be credited to the brain only if it **falls with this local evidence and vanishes without it**. A training-free aggregation meets this criterion on two MEG datasets, raising R@1 by 7.7 and 6.9 points, and yields no spurious gain on EEG, where local evidence is very weak; a stage that uses sentence length in place of evidence fails it. Crucially, source attribution requires no change to a pipeline's task, architecture, or training, suggesting broad applicability across encoder–decoder pipelines. By turning whether a decoder reads the brain from an assumption into a measurement, it lays a foundation for trustworthy non-invasive language decoding. [Project Page](https://whatarewedecoding.github.io/) [Code](https://anonymous.4open.science/r/AuditNeuralDecoding/)