acceptodds
Under review as a conference paper at ICLR 2027

Context-Aware Open-Vocabulary MEG-to-Text Decoding with Large Language Model

Abstract

Decoding magnetoencephalography (MEG) signals into text holds immense potential for restoring communication through non-invasive brain-computer interfaces, but remains highly challenging. The scarcity of paired MEG and text data limits models from learning robust signal to text mappings, causing prior approaches to fall short in the open-vocabulary setting. We address this challenge by reframing open-vocabulary decoding as a context-guided inference task, where MEG signals carry the linguistic content and preceding context guides the language model toward decoding it. From this point of view, we present a novel Context-aware MEG-to-Text decoding framework (CoMET) that leverages preceding context as a semantic anchor, narrowing the textual search space. CoMET first aligns MEG signals with corresponding text at the word-level, and subsequently maps these representations into the latent space of a large language model under contextual conditioning. Extensive evaluations on two MEG-text datasets demonstrate that while prior open-vocabulary decoders collapse to near random performance, CoMET achieves state-of-the-art open-vocabulary decoding performance, highlighting its potential for practical brain-to-text decoding. Code will be made available upon publication.

Primary area:
applications to neuroscience & cognitive science
Keywords:
MEG, Brain-to-Text Decoding, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jspoOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors present an approach for open-vocabulary continuous non-invasive brain-to-text decoding called CoMET. The main innovation in CoMET is to provide the preceding sentence as context to the model before autoregressively predicting the current sentence from the corresponding brain activity. CoMET is trained by aligning corresponding embedding outputs to word embeddings from a frozen text encoder. At inference-time, the model sees sentence-level brain activity. An adapter is also trained to map the output of the encoder to the input of a frozen LLM, which produces the final autoregressive text predictions. The authors test their approach on two datasets and provide ablations against random context and random brain activity controls.

Critical strengths
  • The paper is well-written and generally easy to follow
  • The idea of using prior context is simple and well-motivated
  • Training with a word-aligned contrastive loss over frozen word embeddings, but operating on sentence-level MEG signals at test time is a unique and interesting strategy
  • Table 2 provides a thorough and convincing set of ablations
Critical weaknesses

W1: The main issue with this work is that I do not agree that providing the previous sentence as context is practical or typically available at inference. Providing the previous sentence(s) as context is equivalent to autoregressively predicting the next sentence with teacher-forced tokens for the previous sentence. While I agree that this is better than teacher-forcing every token as some prior work has done, this still does not represent a practical open-vocabulary decoding setting in my view. The authors state that “conditioning on preceding context is practically realizable at inference time” unlike teacher-forcing and that the “preceding context is available at inference”. This is an odd claim to make as teacher-forcing can be thought of as assuming all prior tokens are correctly predicted, and CoMET makes only the slightly weaker assumption that the prior sentence is correctly predicted. I would not say this is “practically realizable” as the decoder needs to have predicted the previous sentence correctly. This makes the comparisons in Table 1 particularly unfair as CoMET is provided information from an oracle for its context.

To overcome this limitation, the authors should test what happens when the previous sentence context comes from the decoder’s own predictions of the last sentence, instead of assuming perfect reconstruction of the previous sentence. How does CoMET perform in this setting?

W2: In section 2.2, the authors introduce their strategy for aligning MEG to text with a contrastive loss between encoded word-aligned MEG and word embeddings from a frozen text encoder. This is very reminiscent of the strategy used in [A] and the authors should cite this work here. At the moment, the paper presents this approach as if it is novel.

W3: LibriBrain contains stories from the canon of Sherlock Holmes and these stories are available in popular public domain text repositories such as Project Gutenberg. As a result, these stories are highly likely to have appeared in the pre-training data of most modern LLMs, leading to a potential data leak. The LLM could use the prior sentence context to continue the story it has already memorised in pre-training. The “Random MEG + Context” ablation in Table 2 somewhat mitigates this concern, so this is only a minor weakness, but I would be keen to understand the authors’ perspective on this problem.

W4: The authors do not provide any uncertainties via training seeds / bootstrap error. It is very difficult to determine which of the results in their tables are significant without some indication of error.

[A] d’Ascoli, Stéphane, et al. "Towards decoding individual words from non-invasive brain recordings." Nature Communications 16.1 (2025): 10521.

Questions for authors

Q1: The authors say in the abstract that the “scarcity of paired MEG and text” is the main limitation for learning signal-to-text mappings. What makes the authors believe MEG provides sufficient information to recover text if MEG-text data were in abundance?

Q2: In 2.2, the authors say a “fixed-size window captures surrounding signals rather than restricting each embedding to the exact word interval”. Does this mean that windows overlap? If this is the case, the authors should be careful about the potential for learning timing shortcuts in training as this can leak the interval between words [B].

Q3: Have the authors considered using LibriBrain100 [C] instead of LibriBrain? It provides almost twice the amount of data and could lead to better results.

Q4: Why do the authors think the length-related effect occurs where increasing M tends to produce longer generated outputs? Did the authors try mitigating this through their prompt? It would otherwise be natural to expect more context to improve decoding performance further.

[B] Jayalath, Dulhan, and Oiwi Parker Jones. "Removing Timing Shortcuts Improves Non-Invasive Brain-to-Text." arXiv preprint arXiv:2609.40359 (2026).

[C] Mantegna, Francesco, et al. "LibriBrain100: One hundred hours of broad and deep MEG data for neural speech decoding at scale." arXiv preprint arXiv:2608.25204 (2026).

13 comments
AuthorsOpenReview ↗

Response to Reviewer jspo (1/3)

We thank the reviewer for the constructive comments. We address each point below.


W1. Practicality of the preceding-sentence context and evaluation with generated history

Thank you for the comment. Teacher forcing provides the ground-truth tokens of the sentence being decoded as input. In contrast, CoMET provides only the complete sentences preceding the sentence being decoded as context, and generates the sentence being decoded from MEG from beginning to end. There are also practical situations in which the preceding sentence is available as text. For example, when reading, the preceding sentence is already given as text, and in dialogue, the interlocutor's previous utterance can be obtained by speech-to-text (STT). Moreover, as the reviewer suggested, when the preceding sentence decoded by the model (generated history) is used as the context for the next sentence, CoMET is lower than with the ground-truth context, but higher on most metrics than the baselines that process each sentence independently, the setting without context (MEG only), and the setting with an irrelevant context (MEG + Random Context). Some baselines achieve higher R-R because of a bias related to output length. Thus, even when the ground-truth preceding sentence is unknown, the approach of CoMET that uses the preceding sentence as context is effective, and its advantage in Table 1 comes not only from access to the ground-truth context but also from the way it decodes by leveraging context. We added this result to Appendix E of the revised manuscript.

LibriBrainBLEU-1BLEU-2BLEU-3R-PR-RR-FWERBERT†
EEG-to-Text9.062.010.4810.1222.5712.60213.820.03
NeuSpeech12.292.780.5910.5712.1211.13114.800.08
MAD6.941.120.0010.927.308.07101.050.06
BrainECHO12.653.000.6213.0616.5613.68133.230.08
MEG only12.803.981.3414.9114.0113.41113.150.10
MEG + Random Context13.203.221.2111.8911.6311.67101.390.13
CoMET (generated history)14.283.131.7314.6814.1214.00103.360.13
CoMET (ground-truth context)16.585.512.2818.2316.7817.3796.350.24

W2. Relation to [A]

Thank you for the comment. The word-level alignment of CoMET is similar to [A] in that both contrastively align word-aligned MEG responses with frozen word embeddings. However, the two methods differ in how windows are processed. [A] segments the raw signal at word onsets and jointly encodes the overlapping windows of a sentence, whereas CoMET first encodes the whole sentence without word-boundary information and then extracts word-level windows from the encoder output and pools each of them independently. Therefore, the timing shortcut raised in Q2 occurs in [A] but structurally does not occur in CoMET (see Q2). In addition, [A] predicts word-level embeddings given word-onset timing and retrieves words from a fixed vocabulary, leaving the combination with language models to future work. In contrast, CoMET uses this alignment only as a training signal for the encoder, and at inference it generates sentences by feeding the whole sentence-level MEG to the LLM as a prefix without word timing. Moreover, the core contribution of CoMET lies not in word-level alignment itself but in sentential context, i.e., leveraging complete preceding sentences as context. To correct the missing citation, we cite [A] in Section 2.2 and state the difference in window processing between the two methods.


[A] Stéphane d'Ascoli, Corentin Bel, Jérémy Rapin, Hubert Banville, Yohann Benchetrit, Christophe Pallier, and Jean-Rémi King. Towards decoding individual words from non-invasive brain recordings. Nature Communications, 16:10521, 2025.

AuthorsOpenReview ↗

Response to Reviewer jspo (2/3)

W3. Possible memorization of Sherlock Holmes

This is a valid concern. We cannot rule out the possibility that the Sherlock Holmes texts were included in the pretraining data of the LLM. However, such memorization does not explain the gains of CoMET. If the LLM were continuing a memorized story from the preceding sentence, the target should be recoverable from context alone without MEG.

(1) Context alone does not recover the target. Given only context, the pretrained LLM achieves a BLEU-1 of 4.08 and a BERT† of −0.07. Even when generation is stopped at the end of the first sentence to match output length, it reaches only 9.21 and 0.11, far below the 16.75 and 0.26 achieved by CoMET under the same condition.

(2) The same holds after training on the training data. The LLM fine-tuned with LoRA on the training split also achieves a BLEU-1 of 3.82 and a BERT† of −0.07, and only 7.22 and 0.12 when stopped at the end of the first sentence. A model that trains an adapter with a dummy prefix using the same LLM and context is also lower than CoMET on all metrics (see our response to Reviewer pBR8, W1 & Q2).

(3) Performance collapses when the MEG comes from another sentence. As in the Random MEG + Context condition mentioned by the reviewer, replacing only the MEG with the signal of another sentence while keeping the ground-truth context drops BLEU-1 from 16.58 to 5.47 and BERT† from 0.24 to 0.05. Even when the MEG is replaced with one that matches the prefix length, the results are a BLEU-1 of 4.17 and a BERT† of 0.06.

(4) Qualitative example. In Table 4(c), CoMET exactly recovers "I have no complaint to make.", whereas Context only, given the same context, generates a completely different sentence. If memorization were responsible, Context only should also have generated the target.

(5) Reproduction on another dataset. GWilliams uses four short fictional stories selected from the Manually Annotated Sub-Corpus (MASC) of the Open American National Corpus as stimuli, which are not as widely known as Sherlock Holmes. On this dataset as well, adding context increases BLEU-1 from 6.08 with MEG only to 10.33 with CoMET, reproducing the effect of context. We added this point to Section 4.3.

In summary, context alone does not recover the target, performance collapses when the MEG is replaced with that of another sentence, and the same effect appears on stories that are not widely known. Therefore, the gains of CoMET come from the combination of sentence information in the MEG and the context, rather than from memorization by the LLM.


Q1. Why MEG would provide sufficient information with more data

Thank you for the question. The following evidence shows that the information that can be decoded from MEG increases with more training data and observations.

(1) The original LibriBrain paper (Özdogan et al., 2025) showed that performance on speech detection, phoneme classification, and word classification improves substantially as the amount of training data increases.

(2) [A] reported that word decoding accuracy increases with the amount of training data and with the number of observations averaged at test time.

(3) In our results as well, although the two datasets differ in subject composition and stimuli, which makes it difficult to attribute the difference to data quantity alone, LibriBrain, with about 52 hours collected from a single subject, achieves higher performance than GWilliams, with about 2 hours per subject.

Therefore, we expect that more linguistic information can be extracted from MEG as data increase. Moreover, CoMET proposes a direction that, even in the current data-limited regime, combines the information in MEG with linguistic context to exploit as much prior information as possible.


[A] Stéphane d'Ascoli, Corentin Bel, Jérémy Rapin, Hubert Banville, Yohann Benchetrit, Christophe Pallier, and Jean-Rémi King. Towards decoding individual words from non-invasive brain recordings. Nature Communications, 16:10521, 2025.

AuthorsOpenReview ↗

Response to Reviewer jspo (3/3)

Q2. Overlapping windows and timing shortcuts

Thank you for the comment. Yes, the windows of adjacent words overlap. However, timing shortcuts do not occur in CoMET.

The timing shortcut in [B] arises when the model jointly receives overlapping windows cropped from the raw signal at each word onset, as in [A]. In this case, the model can infer the interval between words, i.e., word length, by comparing how far the same signal is shifted across windows.

In contrast, CoMET first encodes the MEG of the whole sentence without word-boundary information, then crops word-level windows from the output, pools each of them independently, and compares each only with the embedding of its own word. Although the windows overlap, they are never compared with each other, so timing shortcuts structurally cannot occur. Moreover, Stage 2 decodes sentences from the whole sentence-level MEG without word-level segmentation, so it uses no word timing at all and is unrelated to timing shortcuts. [B] also notes that such leakage is not an issue in settings where word onsets are unknown. We added this explanation to Section 2.2.


Q3. LibriBrain100

Thank you for the suggestion. LibriBrain100 was released on arXiv in August 2026, about a month before the submission deadline, after our experiments were already underway. We believe LibriBrain100 is particularly valuable for this work. With about 80 hours from a single subject, it allows us to examine how performance changes with data quantity, and since it includes stimuli other than Sherlock Holmes, such as podcasts and random sentences, it also allows a direct test of the memorization concern raised in W3. Although we could not use it in this work, we will use LibriBrain100 to examine these questions in future work. Thank you for pointing us to this valuable dataset.


Q4. Effect of M on output length

Thank you for the question.

Why outputs become longer. The average output length increases from 12.6 words for M=1 to 14.6 words for M=2 and 15.0 words for M=3, and the maximum length increases from 60 to 107 words (Section 4.6). Since the MEG prefix is placed at the beginning of the template, a larger M lengthens the text between the MEG and the generation position. We presume that this weakens the influence of the MEG during generation and lets the language prior of the LLM drive the output, producing longer and more elaborate outputs. This is consistent with the trend in Section 4.7, where larger LLMs generate longer outputs.

Effect of more context. Moving from no context (M=0) to M=1 greatly improves performance, and beyond M=1 the performance remains stable regardless of M. On LibriBrain, M=2 and M=3 achieve higher BLEU-1/2/3 than M=1, but the differences are small compared with the difference between M=0 and M=1. That is, a single preceding sentence is sufficient to anchor decoding, and further increasing M does not keep improving performance. We presume the following reasons. First, the preceding sentence is the most directly related to the content of the next sentence, and earlier sentences provide relatively little additional information for narrowing the candidates. Second, as explained above, the relative influence of the MEG weakens as M grows, partly offsetting the benefit of more context. Third, the longer outputs at larger M slightly worsen WER and increase ROUGE-1 recall, so these length effects in the metrics partly mask the improvement in content recovery.


W4. Uncertainty estimates

Thank you for the comment. We agree that uncertainty estimates are necessary to determine which results are significant. We are currently training CoMET and the word-level and sentence-level alignment variants with three seeds, retraining both Stage 1 and Stage 2 for each seed, and computing paired bootstrap 95% confidence intervals of the differences between methods (see also our response to Reviewer pBR8, Q6). We will share the results as soon as they are available.


[A] Stéphane d'Ascoli, Corentin Bel, Jérémy Rapin, Hubert Banville, Yohann Benchetrit, Christophe Pallier, and Jean-Rémi King. Towards decoding individual words from non-invasive brain recordings. Nature Communications, 16:10521, 2025.

[B] Dulhan Jayalath and Oiwi Parker Jones. Removing timing shortcuts improves non-invasive brain-to-text. arXiv preprint arXiv:2609.40359, 2026.

Reviewer_pBR8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CoMET investigates sentence-level reconstruction of perceived speech from MEG using word-level contrastive alignment and a learned prefix to a frozen language model conditioned on the preceding sentence. The framework uses word-level annotations during alignment training but does not require them at inference. Experiments on LibriBrain and GWilliams compare four existing decoders and examine context, alignment granularity, context length, and language model size. The reported lexical and semantic results support further investigation of this combination.

Critical strengths

Context conditioning produces consistent improvements on both datasets. BLEU-1 increases from 12.80 to 16.58 on LibriBrain and from 6.08 to 10.33 on GWilliams when preceding context is included (Tables 2 and 5). These comparisons provide useful empirical evidence for incorporating contextual information into MEG-to-text generation.

Word-level alignment improves over sentence-level alignment on both datasets (Tables 3 and 6), while inference operates without word-level timestamps. This supports the usefulness of local lexical supervision for learning representations that can subsequently support sentence-level generation.

Critical weaknesses

The controls do not yet convincingly isolate recovery of target-specific neural content. Tables 2 and 5 report context-only WER values of approximately 687–718%, compared with 96–111% for CoMET, raising concerns about differences in output length and stopping behavior. The random-MEG comparison is informative, but duration matching is unspecified, and semantic scores are not reported for these input ablations.

The methodological contribution relative to closely related work needs clearer substantiation. BrainLLM already optimizes learned brain prefixes through continuation likelihood while conditioning a frozen language model on preceding text, and its reconstruction using generated history is omitted from Appendix A.3. "Semantic reconstruction of continuous language from MEG signals" also combines contrastive reconstruction of continuous word embeddings from MEG with language-model-guided beam search, although it requires word timing at inference. CoMET’s full-sentence setting and alignment strategy may still be useful, but the broad first-use-of-context claim and comparisons against sentence-independent baselines do not adequately establish this added value.

Questions for authors
  1. In Section 3.1, do sentence indices identify unique stimuli or individual recordings? Since GWilliams repeats stories across participants and sessions, please clarify whether all presentations of each stimulus sentence are grouped before splitting, and provide unique-sentence counts and overlap checks covering both training stages and all evaluated context lengths.

  2. Do the context-only and random-MEG conditions use comparable prompts, generation limits, and stopping rules, and do MEG replacements preserve duration and prefix length? Please strengthen the comparison with a trained context-only control using a learned dummy prefix and report output lengths and BERT† scores for the input ablations. These results would help determine whether the observed differences reflect target-specific neural information or changes in generation behavior.

  3. What improvement remains over a BrainLLM-style adapter when the Qwen backbone, preceding context, and dataset splits are held fixed? Please clarify whether the sentence-level alignment ablation already provides an equivalent comparison or whether important architectural and training differences remain. The distinction between full preceding sentences, partial stimulus context, and generated history should also be reflected accurately in the related-work discussion.

  4. Section 2.1 assumes access to the exact preceding stimulus sentence, while the conclusion discusses restoring communication. What application scenario supplies this transcript, and how would the system operate when previous sentences must themselves be decoded? A short evaluation using generated sentence history would test this dependency; the current audiobook experiments establish perception decoding with supplied textual context.

  5. Section 2.2 defines MEG word windows with shape (d\times W), but does not specify the pooling, projection, and similarity operations used to compare them with (g(s_j)) in Equation (1). Please provide these operations, the encoder’s temporal stride, and the timestamp mapping needed to interpret the effective duration of (W=128). How are repeated lexical items within a contrastive batch handled, given that the stated objective treats every other pair as negative?

  6. Section 3.2 states that all experiments use one fixed random seed, while Tables 3 and 6 provide no uncertainty estimates. Please report paired confidence intervals for the central context and alignment comparisons, accounting for dependence among repeated stimulus presentations, and check training variability for the main word-versus-sentence alignment comparison with a few seeds. This would establish how consistently the observed gains survive changes in training and evaluation samples.

8 comments
AuthorsOpenReview ↗

Response to Reviewer pBR8 (1/4)

We thank the reviewer for the constructive comments. We address each point below.


W1 & Q2. Length/stopping effects and controls for generation behavior

Thank you for the comments. Since W1 and Q2 both ask whether the observed differences reflect target-specific neural information or differences in generation behavior, we answer them together.

All conditions in Table 2 use the same prompt template, except for the MEG prefix, and the same generation configuration (beam search, repetition penalty, no-repeat n-gram, maximum number of tokens, and stopping rule). Nevertheless, conditions without MEG or with the MEG of another sentence often keep generating beyond a single sentence, which lengthens the outputs and greatly increases WER through insertion errors. As the reviewer pointed out, the WER differences in Table 2 mainly reflect output length. To separate the effect of neural information from generation behavior, we added three controls. (1) We applied a stopping rule that terminates generation at the end of the first sentence to the context-only models and CoMET, which matches the output lengths. This rule uses no ground-truth information. (2) We replaced the MEG with that of another sentence of similar length to preserve the prefix length. For this, we used the 1,032 of 3,212 LibriBrain test sentences for which such a replacement is available (target length of 9.31 ± 1.71 words). (3) We trained a learned dummy prefix shared across all samples in place of MEG, together with an adapter of the same architecture as CoMET, using the same data and training configuration. This model can learn the style and output length as CoMET does, but receives no sentence-specific information.

LibriBrain (full test set, first-sentence stopping)BLEU-1BLEU-2BLEU-3R-PR-RR-FWERBERT†#Tokens
Context only (pretrained)9.211.870.4811.3611.2310.12121.900.1117.3
Context only (LoRA)7.222.721.4116.9711.2011.93104.700.1210.4
CoMET16.755.902.6118.7716.5516.5496.740.2616.3
Ground truth––––––––16.9
LibriBrain (1,032 length-matched sentences)BLEU-1BLEU-2BLEU-3R-PR-RR-FWERBERT†#Tokens
Random MEG (length-matched)4.170.820.227.4613.348.34278.440.0643.27
Context only (dummy prefix)10.332.491.0413.4013.3712.95110.670.1410.48
CoMET15.714.292.0416.4115.5716.41100.670.2011.12

Even with matched output lengths, the context-only models fall far behind CoMET, and their BERT† is less than half that of CoMET. The length-matched Random MEG produces long outputs that favor recall, yet its R-R remains lower than that of CoMET, and its BERT†, which is less sensitive to output length, is only 0.06. The dummy prefix control, whose output length is comparable to that of CoMET, is also lower than CoMET on all metrics. Therefore, the gains of CoMET stem from sentence-specific information in the MEG rather than changes in generation behavior. We added the BERT† of the input ablations and the above results to Appendix D of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer pBR8 (2/4)

W2 & Q3. Contribution relative to BrainLLM and Wang et al. (2024)

Thank you for the comments. Since W2 and Q3 both concern the contribution of CoMET relative to BrainLLM and Wang et al. (2024), we answer them together.

Related work and the "first" claim. On Pereira, BrainLLM provides the beginning of the same sentence as the prompt. On Huth and Narratives, the low temporal resolution of fMRI (a TR of 1.5–2 s, with BOLD responses delayed by several seconds) makes it difficult to segment brain signals recorded during continuous listening at sentence boundaries, so the prompt is composed of TR units regardless of sentence boundaries. Therefore, the beginning of the sentence being decoded is usually given in the prompt in this case as well, which is effectively the same setting as Pereira. The original Appendix A.3 described this main setting, and full-text reconstruction is an application that extends it autoregressively. Based on this, and as suggested in Q3, we reorganized the context used in prior work in Appendix A.3 as follows. We also narrowed the broad claim of "the first study to leverage context" to "the first study to leverage sentential context, i.e., complete preceding sentences."

Text given as contextStudies
Beginning of the sentence being decoded (ground truth)BrainLLM
Words previously generated by the modelTang et al. (2023), Wang et al. (2024), full-text reconstruction of BrainLLM
Complete sentences preceding the sentence being decodedCoMET

Comparison with sentence-independent baselines. To verify that the performance of CoMET is not explained solely by access to the ground-truth context, we evaluated it with generated history, which uses the preceding sentence decoded by the model instead of the ground-truth preceding sentence. Even in this setting, CoMET outperforms the baselines that process each sentence independently on most metrics. Some baselines perform better on R-R and WER because of a bias related to output length. We added this result to Appendix E of the revised manuscript.

LibriBrainBLEU-1BLEU-2BLEU-3R-PR-RR-FWERBERT†
EEG-to-Text9.062.010.4810.1222.5712.60213.820.03
NeuSpeech12.292.780.5910.5712.1211.13114.800.08
MAD6.941.120.0010.927.308.07101.050.06
BrainECHO12.653.000.6213.0616.5613.68133.230.08
CoMET (generated history)14.283.131.7314.6814.1214.00103.360.13

Comparison with a BrainLLM-style adapter. BrainLLM maps brain features into the prefix space of the LLM with an adapter and trains it with the continuation likelihood, which is similar to Stage 2 of CoMET. The difference is that CoMET first aligns the MEG encoder with text in Stage 1. The sentence-level alignment variant in Table 3(a) keeps Stage 2 unchanged and only changes the alignment unit of Stage 1 to sentences. Thus, while not identical to a BrainLLM-style adapter, it provides a similar comparison. This variant is lower than word-level alignment on all metrics, which shows that word-level alignment brings additional improvement.

LibriBrainBLEU-1BLEU-2BLEU-3R-PR-RR-FWERBERT†
Sentence14.334.772.1116.1315.3015.11104.830.18
Word (CoMET)16.585.512.2818.2316.7817.3796.350.24

Relation to Wang et al. (2024). Wang et al. and CoMET are similar in that both align MEG with word embeddings through contrastive learning and combine it with a language model. However, Wang et al. require word timing at inference and select, post hoc, among candidates proposed by the LM using MEG embeddings. In contrast, CoMET requires no word timing at inference and directly conditions generation by feeding the sentence-level MEG to the LLM as a prefix. Moreover, Wang et al. use no context other than the LM's own outputs, whereas CoMET additionally provides complete preceding sentences as context. We cite Wang et al. (2024) in Appendix A.3.

AuthorsOpenReview ↗

Response to Reviewer pBR8 (3/4)

Q1. Data split and overlap checks

Sentence indices refer to unique stimulus sentences rather than individual recordings. In GWilliams, the same sentence is presented repeatedly across participants and two sessions, so we grouped all presentations of a sentence, across all participants and sessions, and assigned them to the same split before splitting. Following the sentence order of each story, we used the first 80% for training, the next 10% for validation, and the last 10% for testing, and both training stages (MEG–text alignment and adapter training) use the same split. We also checked for cases where the same text recurs by chance at different positions and removed validation and test sentences that overlap with the training set.

GWilliams (M=1)Unique sentencesOverlap with training setFinal
Train530–530
Validation663 (removed)63
Test68068

Regarding context length, for every evaluated M (1, 2, 3), we excluded the M sentences adjacent to each split boundary from the validation and test sets so that the context of validation and test sentences contains no training sentence. These details are added to Section 3.1.


Q4. Application scenario and generated history

Thank you for the comment. It is correct that the current experiments are perception decoding during audiobook listening. To our knowledge, there is not yet a large-scale public MEG–text dataset collected in settings where preceding sentences naturally continue, such as dialogue or continuous reading, so we used audiobook listening datasets in which sentences continue sequentially. Perception decoding is also a standard setting widely used in non-invasive brain-to-text research [1, 2, 3]. Furthermore, the goal of this work is not limited to restoring communication but lies in open-vocabulary brain-to-text decoding from non-invasive brain signals itself. Accordingly, we revised the conclusion, which emphasized restoring communication.

There are practical situations in which the preceding sentence is available as text. For example, when reading, the preceding sentence is already given as text, and in dialogue, the interlocutor's previous utterance can be obtained by speech-to-text (STT). When the preceding sentence is unknown, the preceding sentence decoded by the model (generated history) can be used as the context for the next sentence. To verify this, we evaluated a generated-history setting in which the first sentence of the test set is decoded without context and each subsequent sentence uses the sentence just decoded by the model as context. Generated history is lower than the setting with the ground-truth context, but higher than the settings without context (MEG only) and with an irrelevant context (MEG + Random Context) on most metrics. That is, context decoded by the model also helps decode the next sentence. We added this result to Appendix E of the revised manuscript.

LibriBrainBLEU-1BLEU-2BLEU-3R-PR-RR-FWERBERT†
MEG only12.803.981.3414.9114.0113.41113.150.10
MEG + Random Context13.203.221.2111.8911.6311.67101.390.13
CoMET (generated history)14.283.131.7314.6814.1214.00103.360.13
CoMET (ground-truth context)16.585.512.2818.2316.7817.3796.350.24

[1] Alexandre Défossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, and Jean-Rémi King. Decoding speech perception from non-invasive brain recordings. Nature Machine Intelligence, 5(10):1097–1107, 2023.

[2] Jerry Tang, Amanda LeBel, Shailee Jain, and Alexander G. Huth. Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroscience, 26(5):858–866, 2023.

[3] Miran Özdogan, Gilad Landau, Gereon Elvers, Dulhan Jayalath, Pratik Somaiya, Francesco Mantegna, Mark Woolrich, and Oiwi Parker Jones. LibriBrain: Over 50 hours of within-subject MEG to improve speech decoding methods at scale. arXiv preprint arXiv:2506.02098, 2025.

Reviewer_CsFUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether MEG recordings acquired during naturalistic listening can support generation of unseen sentences when the preceding sentence is available as context, without access to ground-truth target tokens at inference.

The authors claim two main contributions:

  • A context-guided formulation of open-vocabulary MEG-to-text decoding that uses preceding text to constrain plausible continuations while retaining MEG as evidence for the target content.
  • A two-stage alignment strategy that combines word-level MEG–text supervision with context-conditioned adaptation to a frozen large language model.

The key mechanism is to learn locally informative MEG representations through word-level contrastive alignment, then train an adapter to express these representations as continuous LLM input embeddings so that generation jointly exploits neural evidence and preceding context.

Critical strengths
  1. Strength: The input ablations provide meaningful evidence that decoding benefits from both target-matched MEG signals and relevant preceding context, rather than contextual language generation alone.
    Evidence: On LibriBrain (Table 2), BLEU-1 decreases from 16.58 to 5.47 when MEG is replaced with an unrelated recording, to 12.80 when context is removed, and to 13.20 when context is randomized; context-only generation achieves 4.08. The same qualitative pattern holds on GWilliams (Table 5).
    Why it matters: The paper helps clarify when and how context is useful in this MEG setting, contributing an interpretable design principle beyond reporting a stronger decoder.
  2. Strength: The paper develops a coherent empirical argument for its central hypothesis by examining complementary explanations for the observed gains.
    Evidence: The evaluation connects overall decoding performance (Table 1) with semantic similarity (Figure 3), input-dependence controls (Tables 2 and 5), and alignment-granularity ablations (Tables 3a and 6a), addressing whether improvements reflect meaningful content recovery, dependence on the supplied signals, and benefits from the proposed supervision.
    Why it matters: This convergence of evidence makes the central empirical claim more credible than benchmark improvements alone, supporting the paper’s value as a carefully investigated incremental contribution.
Critical weaknesses
  • Weakness: The novelty of CoMET relative to prior context-based brain-to-text methods could be more clearly established.
    Evidence: The introduction claims to be the first to use context for open-vocabulary brain-to-text decoding, but BrainLLM already combines brain embeddings and preceding text as inputs to a frozen LLM. Appendix A.3 describes BrainLLM as completing a sentence given its beginning, but this only covers part of its experiments: on Huth and Narratives, it uses text from earlier time windows. CoMET’s complete-sentence MEG setting and word-level supervision are meaningful differences, but their contribution beyond prior work would benefit from a clearer explanation.
    Why it matters: This affects the central contribution. The results suggest a useful extension of existing methods, but the current presentation leaves some uncertainty about whether the conceptual advance is substantial enough to support acceptance.
Questions for authors

(A) Clarification questions
None

(B) Suggestions

  • Develop a deeper account of the complementarity between neural evidence and linguistic context. An interesting direction is to explain why contextual conditioning makes weak neural representations more useful for language reconstruction. Beyond describing context as narrowing the search space, a mechanistic account could clarify how contextual expectations interact with signal-derived information, and what determines whether this interaction supports faithful reconstruction or favors a plausible but unsupported interpretation. This would strengthen the conceptual foundations of context-aware brain decoding.
6 comments
AuthorsOpenReview ↗

Response to Reviewer CsFU (1/2)

We thank the reviewer for the constructive comments. We address each point below.


W1. Novelty relative to BrainLLM

Thank you for pointing this out. Appendix A.3 described only the Pereira setting of BrainLLM, but the Huth and Narratives settings are effectively the same. Because fMRI has low temporal resolution (a TR of 1.5–2 s, with BOLD responses delayed by several seconds), brain signals recorded during continuous listening are difficult to segment at sentence boundaries. BrainLLM therefore splits the text into TR units regardless of sentence boundaries and uses the text of the preceding 1–3 TRs as the prompt. As a result, the beginning of the sentence being decoded is usually included in the prompt, and the task becomes continuing that sentence. In contrast, MEG has millisecond temporal resolution and can be segmented at sentence boundaries, so CoMET provides only the preceding sentences as context and generates the target sentence from MEG from beginning to end. In the revised manuscript, Appendix A.3 describes all settings of BrainLLM, and we narrowed the claim from "the first study to leverage context" to "the first study to leverage sentential context, i.e., complete preceding sentences." The two methods differ fundamentally in the unit and role of the context.

(1) Partial-sentence vs. complete-sentence context. This difference is clearly reflected in the dependence on brain signals. When the brain input is randomly replaced, BrainLLM retains 87.8% of its BLEU-1, whereas CoMET drops to 33.0% (BrainLLM: Huth; CoMET: LibriBrain). Although this is not a direct comparison because the signals and datasets differ, it shows that the performance of BrainLLM comes mainly from the prompt, whereas in CoMET the MEG determines the content of the sentence.

BLEU-1Original brain inputRandomly replaced brain inputRetention
BrainLLM18.9916.6887.8%
CoMET16.585.4733.0%

(2) Word-level supervision. For the same reason, word-level neural responses are difficult to isolate in fMRI. In contrast, CoMET exploits the temporal resolution of MEG to align the encoder at the word level, and it outperforms sentence-level alignment on all metrics in Table 3(a).

LibriBrainBLEU-1BLEU-2BLEU-3R-PR-RR-FWERBERT†
Sentence14.334.772.1116.1315.3015.11104.830.18
Word (CoMET)16.585.512.2818.2316.7817.3796.350.24

We are currently repeating this comparison with three training seeds, retraining both Stage 1 and Stage 2 for each seed, and we will share the averaged results as soon as they are available.

AuthorsOpenReview ↗

Response to Reviewer CsFU (2/2)

Suggestion. Mechanism of the complementarity between neural evidence and context

Thank you for the suggestion. We understand the relationship between context and MEG as follows. Context defines a distribution (prior) over plausible next sentences, and MEG serves as evidence (likelihood) for selecting the correct sentence among these candidates.

(1) Context narrows the candidates, but helps only when it matches the MEG. Because the MEG signal is weak, it is difficult to identify the correct word among the entire vocabulary. Once context narrows the candidates, however, the same signal becomes sufficient to discriminate among them. Indeed, providing context reduces the entropy of the next-token distribution from 5.06 to 4.13 (Section 4.4). This effect appears only when the context matches the MEG. Given the same MEG, the correct context improves all metrics over MEG only, whereas an irrelevant context remains at the level of MEG only. Conversely, pairing the correct context with the MEG of another sentence greatly degrades performance. In other words, context does not produce the answer by itself; it helps the matching MEG signal to be interpreted as the correct sentence.

LibriBrainBLEU-1BLEU-2BLEU-3R-PR-RR-FWERBERT†
MEG only12.803.981.3414.9114.0113.41113.150.10
MEG + Random Context13.203.221.2111.8911.6311.67101.390.13
Random MEG + Context5.471.280.3110.0013.109.81278.300.05
CoMET (MEG + Context)16.585.512.2818.2316.7817.3796.350.24

(2) MEG also adds target-specific information to the internal representations of the LLM. We measured how specifically the last-layer hidden state of the LLM, right before generating the target, is close to the representation of the ground-truth sentence. The score is the similarity to the ground-truth sentence minus the average similarity to the other ground-truth sentences.

MEG onlyContext only (learned dummy prefix)CoMET
Similarity (×10⁻²)2.746.859.00

Context alone already represents part of the target-related information, but adding MEG significantly increases the similarity. A paired Wilcoxon signed-rank test between CoMET and Context only confirms that this difference is significant with p < 1e-8. Thus, context narrows down the topic, and within it, MEG adds target-specific information that cannot be inferred from context alone. We added this analysis to Appendix F.

AuthorsOpenReview ↗

Follow-up on W1: Training seeds for word-level alignment

Follow-up on W1: Training seeds for word-level alignment

As mentioned in our response to W1 (2), we repeated the comparison between word-level and sentence-level alignment with three different seeds, retraining both Stage 1 and Stage 2 for each seed. The table below reports the mean and standard deviation over the three seeds on LibriBrain.

LibriBrain (mean ± std over 3 seeds)BLEU-1BLEU-2BLEU-3R-PR-RR-FWERBERT†
Sentence14.11 ± 0.224.41 ± 0.321.91 ± 0.1915.82 ± 0.2915.16 ± 0.1215.19 ± 0.11105.11 ± 0.420.17 ± 0.01
Word (CoMET)16.53 ± 0.195.54 ± 0.222.40 ± 0.2518.32 ± 0.2416.64 ± 0.1317.44 ± 0.1498.03 ± 1.460.24 ± 0.02

Word-level alignment outperforms sentence-level alignment on all metrics for every seed. In addition, we computed 95% confidence intervals of the difference between the two models reported in the paper with a paired bootstrap, which resamples the test sentences 5,000 times and recomputes each metric on every resample. All intervals exclude zero.

LibriBrain (Word − Sentence)BLEU-1BLEU-2BLEU-3R-PR-RR-FWERBERT†
Difference+2.25+0.74+0.17+2.10+1.48+2.26−8.48+0.06
Paired 95% CI[+1.64, +3.12][+0.42, +1.63][+0.13, +1.06][+1.24, +3.21][+0.15, +2.11][+0.37, +2.58][−8.70, −3.40][+0.01, +0.13]

These results confirm that word-level supervision, which exploits the temporal resolution of MEG, brings a consistent improvement. We will add these results to the revised manuscript.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback. We have revised the manuscript accordingly, and the main changes are summarized below. Revised text is highlighted in blue in the revised manuscript.

  • Scope of the novelty claim (Introduction, Appendix A.3). We narrowed the claim to being the first to leverage sentential context, i.e., complete preceding sentences, for open-vocabulary brain-to-text decoding. In contrast, BrainLLM uses part of the same sentence being decoded as context, which limits its use in practice, since the beginning of the target sentence is not available before decoding. We explain this difference in Appendix A.3.
  • Relation to d'Ascoli et al. (2025) and timing shortcuts (Section 2.2). We cite d'Ascoli et al. (2025) and explain how CoMET differs in window processing: windows are extracted from the encoder output of the whole sentence and pooled independently, so inter-word timing is not exposed.
  • Stage 1 details (Section 2.2, Appendix B.1). We specify the pooling, the similarity used in the InfoNCE loss, the temporal stride, the timestamp mapping, and the handling of repeated words within a batch.
  • Data split (Section 3.1). We clarify that sentence indices refer to unique stimuli, that all presentations of a sentence are assigned to the same split, and how the boundary sentences and duplicated texts are excluded.
  • Controlling for generation behavior (Section 4.3, Appendix D). We add BERT† for the input ablations and three controls: a first-sentence stopping rule, a length-matched random MEG, and a learned dummy prefix.
  • Generated history (Appendix E). We evaluate CoMET with its own decoded preceding sentences as context, so that no ground-truth text is used at inference.
  • Memorization concern (Section 4.3). We note that the effect of context is reproduced on GWilliams, whose stimuli are far less widely known than Sherlock Holmes.
  • Representation-level analysis (Appendix F). We add an analysis of the LLM hidden states showing that MEG adds target-specific information beyond the context.
  • Conclusion. We revised the conclusion to focus on open-vocabulary brain-to-text decoding from non-invasive signals and to discuss reading and dialogue as application scenarios.

In addition, we are currently running experiments with multiple training seeds and paired bootstrap confidence intervals, and we will share the results as soon as they are available.

AuthorsOpenReview ↗

Summary of the discussion

Dear Area Chair,

Thank you for the time and effort you have devoted to our paper. As the discussion period draws to a close, we would like to briefly summarize the progress of the discussion for your consideration.

All three reviewers recognized the value of using preceding context for MEG-to-text decoding. The discussion with each reviewer progressed as follows.


  • Reviewer pBR8 acknowledged the consistent improvements from context on both datasets and the benefit of word-level alignment without requiring word timing at inference. During the discussion, they stated that our responses and additional results had largely addressed their concerns and raised their rating to 3. Their remaining suggestion, a comparison with a BrainLLM-style variant under the same conditions, requires about 30 hours of training and could not be completed within the discussion period, so we will add it to the final version.

  • Reviewer jspo found the paper well written, the idea simple and well motivated, the strategy of training with a word-aligned contrastive loss over frozen word embeddings while operating on sentence-level MEG signals at test time unique and interesting, and the ablations thorough and convincing. During the discussion, they stated that the permutation test gives sufficient confidence that sample-specific information is decoded and that the random generated history control supports the usefulness of generated context, and raised their rating to 3.

  • Reviewer CsFU noted that the input ablations provide meaningful evidence that decoding benefits from both target-matched MEG and relevant context, and that the paper develops a coherent empirical argument for its central hypothesis. During the discussion, Reviewer CsFU found the retention comparison and the hidden-state similarity analysis helpful and considered the clarified claim around sentential context appropriate, but maintained their rating because the retention comparison between BrainLLM and CoMET is not controlled for signal modality and dataset. We clarified that this comparison is not a performance comparison but an analysis of each model's dependence on the brain signal under the same manipulation, and that the high retention of BrainLLM, which provides the beginning of the target sentence as the prompt, is a finding already reported in the original BrainLLM paper. Moreover, the retention comparison serves as supporting evidence for the different role of the context rather than as the basis of our claim. Reviewer CsFU's follow-up was posted about two hours before the end of the discussion period, and the suggested comparison requires about 30 hours of training, so it could not be completed in time. We will add the suggested comparison to the final version.

We have provided responses, analyses, and revisions for all concerns raised by the reviewers:

  • Scope of the novelty claim. We narrowed our claim to the first use of sentential context, i.e., complete preceding sentences, and clarified in Appendix A.3 that BrainLLM provides the beginning of the sentence being decoded as context.
  • Target-specific MEG information. Under identical generation settings, we added a first-sentence stopping rule, length-matched random MEG, a learned dummy prefix, and BERT† scores. In all controls, CoMET remains clearly above the context-driven baselines (Appendix D).
  • Decoding without ground-truth text. With generated history, CoMET significantly outperforms both MEG only and MEG + Random Context over three seeds, and a permutation test and a control with randomly assigned generated histories support this result (Appendix E).
  • Uncertainty estimates. Word-level alignment outperforms sentence-level alignment for every seed, and paired bootstrap confidence intervals of the difference exclude zero for all metrics (Appendix G).
  • Mechanism of complementarity. A representation-level analysis of the LLM hidden states shows that MEG adds target-specific information beyond the context (Appendix F).
  • Remaining requests Both comparisons will be included in the final version. For the retention comparison under a BrainLLM-style prompt on the same MEG data, we expect high retention, as reported in the BrainLLM paper, in contrast to the drop with complete preceding sentences. For the BrainLLM-style variant, we expect CoMET to perform better, since Stage 1 provides explicit word-level contrastive learning between MEG and text, which the variant lacks.

Thank you again to you and all the reviewers for your constructive feedback, which has strengthened the paper. We hope this summary is helpful for your assessment.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.