acceptodds
Under review as a conference paper at ICLR 2027

NeuroLens: Cross-Modal EEG-to-Text Decoding via Structured State Space Encoding and Contrastive Alignment

Abstract

Open-vocabulary EEG-to-text decoding generates the sentence a participant is reading from the neural recording alone, and underpins assistive communication and brain–computer interfaces. Progress on this task has come from stronger EEG encoders paired with a pretrained language decoder, yet once evaluation is made rigorous, with sentence-disjoint splits and no teacher forcing, absolute scores are low and it is unclear how much of each score is read from the brain signal and how much is supplied by the decoder's language prior. We identify three limitations of current systems: the encoders in use are poorly matched to short, noisy, word-level EEG sequences; the token-level objective is satisfied by fluent text whether or not the decoder reads the signal; and the contribution of the language prior is neither controlled nor measured. We propose NeuroLens, in which each component answers one of the above limitations: (1) a bidirectional S4D encoder with HiPPO-initialised long-range memory; (2) a reliance-aware objective that combines shuffled-EEG discrimination, contrastive alignment and a vocabulary prior, so that dependence on the paired recording is rewarded directly; and (3) decoder prior dampening, a tunable coefficient on decoder self-attention, together with a verification protocol that replaces the EEG input with shuffled, noise and zero signals over three seeds. On ZuCo 1.0 under a sentence-disjoint protocol without teacher forcing, NeuroLens reaches BLEU-1 of 22.44 and BLEU-4 of 1.46, improving on the previous state-of-the-art by 92% on BLEU-4, and its BLEU-4 falls by 68% when the EEG is shuffled across sentences and by 89% when it is zeroed, which shows that the outputs depend on the recorded signal. A matched-protocol ablation and a dampening sweep identify which components carry the gain and which do not. Code is available at https://github.com/anonymous399764/AnonymousEEG2TEXT.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG-to-text decoding, brain-computer interfaces, structured state space models, S4D, cross-modal alignment, contrastive learning, cross-attention collapse, ZuCo

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_G6aZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem of signal reliance in EEG-to-text decoding, where pretrained language models may generate text without sufficiently utilizing EEG signals. The authors propose NeuroLens, which combines a bidirectional S4D encoder, reliance-aware training, and decoder prior dampening to improve text generation and encourage greater dependence on EEG signals.

Critical strengths
  1. The paper addresses an important issue in EEG-to-text decoding and adopts a problem-driven approach, identifying limitations of existing methods and designing specific components to address them
  2. The paper is well-structured and logically organized, making the motivation, methodology, and experimental results easy to follow.
  3. The paper adopts a relatively rigorous evaluation protocol, including sentence-disjoint splits, decoding without teacher forcing, and shuffled/zero EEG controls to assess the model's dependence on neural signals.
  4. NeuroLens improves BLEU-4 from 0.76 to 1.46 over the strongest reported baseline, while shuffled EEG reduces BLEU-4 by 68%, indicating sensitivity to the paired EEG input.
Critical weaknesses
  1. The paper mainly compares against a limited set of existing baselines, while recent works such as [1] and [2] also investigate EEG-grounded text generation and reliance on language priors. A more comprehensive discussion and, where evaluation protocols permit, quantitative comparisons would help clarify the contribution of NeuroLens.
  2. The ablation results do not convincingly establish the contributions of individual components. In Table 3, an independent full-model run shows a 21% decrease in BLEU-4, exceeding the reported gains from several proposed modules. Moreover, removing InfoNCE, word dropout, or the attention gate improves performance. These results raise concerns about the stability of the reported improvements and the necessity of certain components.
  3. Although shuffled-EEG experiments demonstrate input sensitivity, they do not necessarily establish reliance on sentence-specific semantic information. Additional controls, such as length-matched and within-subject EEG shuffling, would strengthen this claim.
  4. Setting decoder self-attention to zero is too extreme and may limit the decoder's ability to utilize linguistic context.

Reference

[1] Liu, Xiaozhao, Dinggang Shen, and Xihui Liu. "Learning interpretable representations leads to semantically faithful eeg-to-text generation." arXiv preprint arXiv:2505.17099 (2025).

[2] Wang, Yuchen, et al. "Escaping the BLEU Trap: A Signal-Grounded Framework with Decoupled Semantic Guidance for EEG-to-Text Decoding." arXiv preprint arXiv:2603.03312 (2026).

Questions for authors
  1. Could the authors clarify the differences between NeuroLens and recent EEG-to-text methods such as GLIM and SemKey, and explain whether comparisons under a unified evaluation protocol are feasible?
  2. Could the authors further clarify the effectiveness of individual components, given the substantial BLEU-4 variation between the two full-model runs in Table 3?
  3. How do the authors account for potential confounding factors, such as sequence length and subject-specific differences, in the shuffled-EEG experiments?
1 comment

Since there are no response from the authors, I will maintain my original score.

Reviewer_KsEcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NeuroLens for generating reading text from word-level EEG features. The method combines a bidirectional S4D encoder, training objectives including shuffled-EEG discrimination and cross-modal alignment, and decoder self-attention scaling, aiming to reduce the model's reliance on language priors. The authors report performance gains on ZuCo 1.0 under sentence-non-overlapping, no-teacher-forcing evaluation, and test signal dependence through shuffled, noisy, and zeroed inputs.

Critical strengths
  1. The research question is valuable. The paper directly asks whether the generative model truly uses EEG rather than relying only on a pretrained language model to produce fluent output, which is important for judging real progress in EEG-to-text.

  2. The main evaluation protocol is fairly rigorous. The authors use sentence-non-overlapping splits, free generation, and normalization statistics estimated only from the training set, and report main results over three training seeds.

  3. It provides signal-dependence tests and fairly transparent experimental materials. Input replacement controls are more informative than simply reporting generation scores; the released code, logs, and ablations of components that brought no gain also make the method easier to check further.

Critical weaknesses
  1. Because the loss on shuffled input is gradient-truncated, when the hinge is active, this term provides a gradient equal to an extra positive-sample cross-entropy gradient. Therefore, the existing ablation cannot distinguish whether the gain comes from paired-relation discrimination or from cross-entropy reweighting. The paper lacks a corresponding equal-weight control and an analysis of hinge activation rates.

  2. In Table 3, the full model and its repeat run have BLEU-4 of 0.96 and 0.76, respectively, a fluctuation larger than some reported component gains. Removing InfoNCE, the attention gate, or using a linear projection even improves BLEU-4. Single-seed ablations are not enough to confirm the necessity of the full design, and the explanation of "keeping components to improve stability" also lacks direct evidence.

  3. The 0.9 on ZuCo 2.0 and the baseline's 0.76 on ZuCo 1.0 come from different test sets, so they cannot be used to judge the method's generalization advantage. The subject-wise experiments also do not clearly ensure sentence non-overlap at the same time, so double-isolated evaluation and baselines under the same test conditions are needed.

Questions for authors
  1. Can you add shuffled controls matched on sentence length, valid positions, subject, and reading task, as well as a control that preserves word boundaries but replaces EEG values? Would the authors be willing to narrow "contribution proportion" to "performance change caused by input replacement"?

  2. Can you add an equivalent cross-entropy reweighting baseline and report the hinge activation rate and the loss difference between real and shuffled inputs, to verify the independent role of the discriminative loss?

  3. Can you repeat multi-seed ablations under the checkpoint selection protocol used in the main experiments, and perform paired bootstrap by independent sentences to report confidence intervals for component differences?

  4. Can you evaluate under a setting where both subjects and sentences are non-overlapping, and have the main baselines perform zero-shot transfer comparison on the same ZuCo 2.0 test set?

  5. Beyond n-gram overlap, can you evaluate recovery accuracy for entities, dates, and relations, and clarify the respective roles of word-level segmentation information from eye tracking and EEG values?

Reviewer_9Z5zOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes NeuroLens, an EEG-to-Text decoding framework that combines a BiS4D Encoder, a Reliance-Aware Objective (RAO), and Decoder Prior Dampening (DPD). The goal is to encourage the model to rely more on EEG signals rather than language priors.

Critical strengths

S1. The paper addresses an important issue in EEG-to-Text decoding: models often rely too heavily on language priors while overlooking actual neural signals. The proposed RAO is a valuable direction to explore.

S2. The paper includes control experiments with Shuffled/Noise/Zero EEG inputs, providing some evidence for evaluating the model's signal reliance.

Critical weaknesses

W1. The BiS4D Encoder mainly adopts the existing S4D architecture, with limited EEG-specific innovation. Results in Table 3 also show no clear advantage. While it is a reasonable backbone choice, its contribution seems somewhat overstated.

W2. The RAO appears to be a simple combination of auxiliary losses, without sufficient justification for its design. Moreover, the current experiments do not convincingly demonstrate improved signal reliance.

Questions for authors

Q1. BiS4D mainly builds on an existing architecture, and a simple Linear Baseline remains competitive (table 3). Is there sufficient novelty to justify it as a standalone contribution?

Q2. Since the negative-sample loss is detached in Shuffled-EEG Discrimination (line 251), does it actually provide an optimization signal for signal reliance beyond standard CE loss?

Q3. Since the paper aims to reduce reliance on language priors, BLEU improvements alone are not sufficient to validate this claim. Could the authors compare the performance drop from Real to Shuffled EEG across the Full Model and its variants (w/o Discrimination, w/o Vocabulary Prior, and CE Only)?

Q4. In Table 3, the replicated run yields a BLEU-4 score 21% lower than the originally result, which may even exceed the gains from several proposed components. Could the authors clarify the source of this discrepancy?

Overall, the paper addresses a worthwhile research question, but the method and experimental design are not yet sufficiently centered around it. Some experimental results also raise concerns. I would be open to raising my score if these concerns are adequately addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.