acceptodds
Under review as a conference paper at ICLR 2027

Escaping the BLEU Trap: A Signal-Grounded Framework with Decoupled Semantic Guidance for EEG-to-Text Decoding

Abstract

Decoding natural language from non-invasive EEG signals is a promising yet challenging task. However, current state-of-the-art models remain constrained by three fundamental issues: Semantic Bias, where outputs collapse into generic linguistic templates; Signal Neglect, where models rely heavily on LLM priors to hallucinate fluent text even in the absence of meaningful signals; and the "BLEU Trap", where high-frequency stopwords inflate n-gram metrics, masking a lack of true semantic fidelity. To resolve these challenges, we move beyond conventional end-to-end pipelines and propose SemKey, a novel two-stage framework that enforces signal-grounded generation through four decoupled semantic objectives: sentiment, topic, length, and surprisal. We extract these semantic anchors from EEG embeddings directly, then unify them with an Active Retrieval Decoding mechanism, encouraging the LLM to ground its token generation in the neural signals rather than defaulting to linguistic priors. Furthermore, we break the BLEU Trap by establishing a comprehensive evaluation protocol using rigorous retrieval and distribution-based metrics such as Fréchet Distance. Extensive experiments under fully autoregressive inference, including noise and time-shuffled EEG controls, provide evidence of signal dependency and demonstrate that SemKey achieves SOTA performance on these robust protocols. Code will be released upon acceptance.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Brain-Computer Interface (BCI), Electroencephalography (EEG), EEG-to-Text, Large Language Models (LLMs), Semantic Guidance

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NH1QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SemKey, a two-stage EEG-to-text system built on a GLIM-like EEG–language alignment backbone. Stage 1 predicts four attributes (topic, sentiment, length and surprisal) alongside EEG–text representation learning. Stage 2 turns the predicted attributes into a fixed prompt and supplies EEG states as keys and values for cross-attention in a frozen Flan-T5 decoder. Beyond BLEU, it evaluates fully autoregressive outputs using sampled N-way retrieval, sentence-embedding similarity, diversity and distributional metrics. SemKey improves these metrics over GLIM and exhibits stronger changes under Gaussian-noise and time-shuffled EEG inputs.

Critical strengths

The paper takes the teacher-forcing and BLEU evaluation problem seriously. It reports autoregressive generation and compares retrieval, SBERT, Self-BLEU, head entropy and Fréchet distance rather than treating n-gram overlap as neural-decoding accuracy. The MTV versus standard-reference BLEU comparison (Table 2) and prefix-stripping analysis (Appendix G) give concrete evidence of the risks of rewarding generic text templates.

The improvements over GLIM are measurable, and the ablations go beyond a single headline score. In Table 1, 24-way retrieval improves from 10.74% to 16.24%, SBERT from 0.194 to 0.221, and Self-BLEU decreases from 90.86% to 52.08%. The paper reports multi-seed comparisons, the contributions of the attribute prompt and Q-K-V interface (Table 5), and both Gaussian and time-shuffling tests. These provide more evidence of nontrivial conditional behavior than an ordinary BLEU-only method paper.

Critical weaknesses

The reported semantic supervision and retrieval gains do not isolate sentence-specific meaning decoded from EEG; a corpus-metadata shortcut is a particularly concrete confound. Section 3.1 explicitly embeds subject ID and dataset information as priors. Yet Appendix C.1 defines the Topic labels as the two ZuCo corpus categories, “Biographies and Factual Knowledge” versus “Movie Reviews and Sentiment,” which may be highly predictable from corpus or task identity. Table 4's 83.51% Topic accuracy therefore cannot be credited to neural semantic extraction without a metadata-only control. Moreover, the N-way retrieval test samples unrelated reference sentences at random (Section 3.3), permitting category and broad sentence-style differences to help identify the target. At 24-way, accuracy is 16.24%, but the absolute paired SBERT score is only 0.221; examples in Appendix Table q frequently substitute unrelated entities, dates and biographical facts. Attribute-matched hard negatives and recovery of content words and named entities would be needed to support the claimed fine-grained, sentence-specific grounding. The current experiments demonstrate improvement in broad semantic conditioning, not convincing recovery of the particular text being read.

The “signal-grounded” interpretation goes beyond what the controls establish, and the distinctiveness of the mechanism is therefore overstated. The Gaussian-noise and 20-segment time-shuffle tests, together with the ablations of the attribute prompt and the query–key–value attention pathway, show that predictions are sensitive to perturbing the input. However, both controls can move the model away from its training distribution; the absence of coherent output under noise is not evidence that its coherent outputs contain correct neural semantics. The paper never performs a same-checkpoint, in-distribution substitution using a real EEG recording from another sentence, ideally matched on subject, corpus topic and length. This distinction is central because its architecture primarily adds conventional auxiliary supervision, a structured prompt and cross-attention to a GLIM-like framework. Table 5 establishes that these design choices improve retrieval and output diversity, but not whether they improve the contribution of correctly paired EEG rather than guiding the language model with coarse attributes and corpus priors. The text-disjoint split, with subjects shared between training and test (Appendix K), does not resolve this attribution issue.

Questions for authors

What is Topic classification accuracy using only the subject identity, dataset membership, and task information, without EEG? How do the 4-way and 24-way retrieval results change when every distractor shares the target's topic and is matched approximately for length and sentiment?

With a fixed trained checkpoint, what happens to retrieval accuracy, recovery of named entities and content words, and generation quality when each EEG recording is replaced by a real recording from a different sentence of the same participant and corpus category? Can the prompt-only and EEG-value contributions be separated under this in-distribution intervention?

Reviewer_8GMWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines a reliability issue in EEG-to-text decoding: models may rely on language priors and repeatedly generate generic templates while obtaining BLEU scores that do not adequately reflect semantic fidelity. To address this issue, the authors propose SemKey, a two-stage framework that guides generation through explicit attributes. The first stage combines EEG–text representation alignment with multitask learning to predict topic, sentiment, sentence length, and surprisal from EEG. The second stage converts these predictions into textual prompts and combines them with continuous EEG representations through cross-attention to guide a frozen language model, without ground-truth text prefixes at inference. Experiments on ZuCo 1.0 and 2.0 use established retrieval, text similarity, diversity, and distributional metrics. The authors report improvements over the main baseline, GLIM, in semantic matching and output diversity, and investigate component contributions and input dependence through ablations, Gaussian-noise controls, and temporal shuffling.

Critical strengths

S1. The paper addresses a meaningful reliability problem in EEG-to-text decoding. Rather than focusing exclusively on improving BLEU, it examines whether generated outputs are repetitive, match the target semantics, and depend on EEG inputs. The discrepancy between template-based outputs and BLEU scores provides useful empirical evidence of the limitations of relying on lexical overlap alone.

S2. The approach is clearly motivated and yields consistent improvements under the reported evaluation protocol. Converting EEG-predicted attributes into explicit prompts and combining them with continuous EEG representations provides an understandable mechanism for guiding generation. Compared with GLIM, SemKey improves retrieval accuracy across multiple candidate-set sizes, sentence similarity, and output diversity. For example, 4-way and 24-way retrieval accuracy increase by approximately 8.03 and 5.50 percentage points, respectively. Results across independent training runs further support the consistency of these improvements.

S3. The experiments include a substantial range of component analyses and input controls. In addition to overall performance, the authors evaluate attribute prediction, the contributions of individual attributes and decoding components, and the effects of Gaussian noise, temporal shuffling, and prompt perturbations. These analyses provide complementary evidence about the sources of performance and indicate that the outputs are not determined entirely by language-model priors.

Critical weaknesses

Q1. The methodological novelty primarily lies in combining existing components, while the stronger claim of semantic disentanglement remains insufficiently supported.

The approach predicts topic, sentiment, sentence length, and surprisal through multitask learning and converts these predictions into prompts. While this design is reasonable, the granularity of these attributes does not fully support the paper’s characterization. Topic prediction distinguishes only between “Biographies and Factual Knowledge” and “Movie Reviews and Sentiment,” while sentiment prediction distinguishes neutral from non-neutral content without distinguishing positive from negative polarity, which is particularly relevant to movie-review semantics. Sentence length is a structural attribute, and surprisal reflects predictability; neither directly establishes recovery of specific semantic content. These attributes therefore provide coarse semantic or statistical cues, but do not by themselves demonstrate the extraction of sentence-level information about entities, events, or relations. Moreover, multiple prediction heads operating on shared EEG representations do not automatically establish disentanglement. The paper lacks analyses of information overlap between attributes or the ability to selectively manipulate them. Since cross-attention itself is also an established mechanism, the authors should more clearly distinguish multi-attribute supervision, explicit prompt conditioning, and semantic disentanglement, and identify the substantive methodological advance over prior approaches.

Q2. Competitive baseline coverage is limited and primarily centers on GLIM.

Although several prior methods are listed, some results use teacher forcing at inference and are included only for reference. Under the directly comparable generation setting, GLIM remains the main competitive full-model baseline. Comparison with this single methodological approach provides limited evidence that the improvements extend to other EEG–language alignment or multitask generation frameworks. Relevant methods such as BELT-2 and CET-MAE/E2T-PTR merit consideration under consistent data splits, input-information conditions, and inference without teacher forcing, or an explanation of why such comparisons are not applicable. The current results support superiority over GLIM under the reported protocol, but provide less support for a broader state-of-the-art claim.

3. The evaluation protocol is useful, but its contribution beyond established evaluation practices is not sufficiently delineated.

Retrieval evaluation, sentence-embedding similarity, Self-BLEU, and Fréchet distance all have established precedents. Earlier work, including GLIM, has also emphasized semantic evaluation beyond lexical matching and the use of noise controls. Combining these tools to analyze template collapse is a useful extension, but the paper does not sufficiently establish what the protocol adds or why it provides a more reliable assessment of semantic fidelity. In particular, retrieval with randomly sampled candidates may benefit from coarse topic distinctions, diversity does not ensure correctness, and aggregate distributional distance cannot establish correspondence between individual EEG inputs and generated texts. The framing around moving beyond the “BLEU Trap” would therefore benefit from a clearer account of the protocol’s incremental contribution and remaining limitations. If the protocol is presented as a major contribution, its ability to detect within-topic semantic errors and incorrect entities or relations should be validated, rather than inferred from improvements across multiple metrics.

4. The ablations are numerous, but their configurations are insufficiently specified to isolate the sources of improvement.

Table 5 includes the “w/o Q-K-V” and “w/o Prompt” variants. However, it is unclear whether these variants retain all four auxiliary attribute losses, whether removing an attribute also removes its prediction supervision, and what training differences remain when reverting to GLIM-style decoding. These ambiguities make it difficult to distinguish gains from auxiliary supervision, explicit attribute conditioning, and the particular decoding configuration. Furthermore, replacing prompts or EEG at inference tests the trained model’s dependence on these input pathways, but does not substitute for an independently trained attribute-only generation baseline. Such a baseline is particularly important for determining whether continuous EEG representations provide useful information beyond the coarse predicted attributes.

5. The input-dependence controls do not fully identify what information the model uses, and the scope of generalization remains limited.

Performance degradation under Gaussian noise and temporal shuffling supports dependence on EEG inputs. However, these controls alone cannot distinguish the use of sentence semantics from reading duration, task differences, or other input statistics. Since the topic labels are closely tied to corpus source, more targeted controls are needed, such as retrieval within matched topics, tasks, and length ranges, or controlled substitutions of real EEG representations between matched samples. In addition, the experiments are restricted to ZuCo 1.0 and 2.0 and do not establish generalization to unseen subjects. Claims about broader EEG-to-text applicability should be qualified accordingly.

6. The retrieval improvements may partly reflect coarse attribute matching rather than finer semantic recovery.

SemKey explicitly learns topic, sentiment, and length, and retrieval with randomly sampled candidates may reward precisely these distinctions. For example, when the target is a biography and the candidate set contains movie reviews, a generated sentence may identify the correct candidate through broad topical similarity without recovering the relevant person or event. This is not evidence of data leakage, but it makes improvements from attribute conditioning difficult to distinguish from more accurate sentence-level semantic recovery. Evaluation among candidates matched for topic, sentiment, and length, together with analyses of entity, event, or relation correspondence, would help resolve this ambiguity.

7. Reduced template repetition does not yet establish reduced semantic hallucination.

The reported improvements in diversity are clear, but more varied outputs may still introduce details unsupported by the EEG input. Some qualitative examples in the supplementary material retain the general form of a biography or movie review while failing to recover specific identities, dates, or events accurately. The current evidence therefore does not fully distinguish more accurate generation from generation that is more diverse and topically appropriate. A blinded assessment or fine-grained content evaluation that separately measures correctly recovered information and unsupported additions would provide stronger evidence for the claimed improvement in semantic fidelity. Minor concerns

Minors:

The reference baseline for sentiment prediction requires further justification. A 50% chance level is appropriate for uniform random guessing, but comparison with this value alone may be uninformative if neutral and non-neutral examples are imbalanced. Class proportions, a majority-class baseline, and balanced accuracy or macro-F1 should be reported.

The definition and interpretation of surprisal also require clarification, including the language model, tokenization, and whether the attribute represents total or average surprisal. Total surprisal may correlate with sentence length; without examining this relationship, it is difficult to determine whether it provides distinct information. This also affects the claims of attribute complementarity and disentanglement.

Questions for authors

Q1. Does SemKey retain its advantage when coarse attributes are controlled?

Could the authors evaluate semantic retrieval using candidate sets matched for topic, sentiment, and sentence length, and compare the results with GLIM? This would help determine whether the gains extend beyond broad topical and statistical matching to more specific information about entities, events, and relations.

Q2. Do continuous EEG representations provide useful information beyond the four predicted attributes?

Could the authors independently train a generator conditioned only on the predicted attributes, without continuous EEG representations, and compare it with the full model? Inference-time input replacement may introduce distribution shift and does not fully establish how much performance can be explained by the attributes alone.

Q3. Does reduced repetition translate into more accurate content and fewer semantic hallucinations?

Could the authors conduct a blinded assessment or fine-grained content evaluation of SemKey and GLIM, separately measuring correctly recovered information, omissions, and unsupported entities, dates, or events? This would distinguish more varied generation from more faithful generation.

Q4. Does the continuous EEG pathway carry sample-specific information beyond the attribute prompt?

Could the authors hold the original attribute prompt fixed while replacing the continuous EEG representations with those from another sample matched for subject, task, topic, and approximate length? Comparing the resulting output’s correspondence to the original and replacement references would help assess whether this pathway contributes sample-specific content beyond the coarse attributes.

Reviewer_SCgnOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SemKey, a teacher-forcing-free EEG-to-text decoder for the ZuCo datasets that extends GLIM. Its EEG encoder is trained with auxiliary heads that predict topic, sentiment, sentence length and surprisal. These predictions are written into a text prompt for a frozen Flan-T5 decoder, which attends to the EEG embeddings through cross-attention. The authors also argue that BLEU rewards templated outputs (the "BLEU Trap") and propose retrieval, content-recall, diversity and Fréchet-distance metrics instead.

Across seven seeds, SemKey beats GLIM on these metrics (4-way retrieval 50.4% vs. 42.4%) but scores lower on BLEU. Given Gaussian noise as input, it outputs gibberish rather than fluent text.

Critical strengths

Timely, evidence-backed critique of the current evaluation protocol. After Jo et al. (2024) showed that teacher forcing inflated earlier results, GLIM's MTV-BLEU became the de facto replacement. This paper shows that this protocol can also be gamed. Removing MTV references or stripping common prefixes cuts GLIM's BLEU-2 by about 88% and 39% respectively (Tables 2 and j). This matters because it should influence how the field reports results, independent of whether SemKey itself holds up.

It measures the prior state of the art's output collapse, and the method clearly fixes it. Scoring diversity against the ground-truth values, rather than rewarding extreme values, turns "GLIM repeats templates" into a measured result. GLIM's Self-BLEU is 90.9 and its head entropy 2.18, against 49.4 and 6.35 for the ground-truth text. SemKey reaches 52.1 and 6.21 with low seed variance, and Table m shows the difference qualitatively. These metrics cannot certify faithful decoding (Weakness 2), but they reliably detect collapse, and removing a known failure of the strongest baseline is a real, if partial, advance.

Critical weaknesses
  1. The four "semantic" hints may come from shortcuts rather than brain signals. The model guesses topic, sentiment, length and surprisal, then feeds these guesses to the language model as hints. Each can be guessed without reading the EEG. Topic is just whether the sentence is a movie review or a Wikipedia biography, and in ZuCo that is almost fully determined by which recording session it came from, information the model is given directly. Sentiment mostly follows topic, because only the movie reviews carry sentiment. Length can be read from how long the recording lasts, since longer sentences take longer to read. Surprisal differs between reviews and biographies in general. The hint accuracies are also compared against a 50% coin flip rather than against always guessing the most common label. This matters because scrambling the hints (App. F) drops 4-way retrieval from 50.5% to 36.1%, below GLIM, so most of the improvement runs through them. If they come from shortcuts, the central claim that SemKey decodes meaning from EEG is not supported.

  2. The replacement metrics can be gamed in the same way the paper says BLEU is. None of them confirms that an output matches the specific sentence the subject read. Retrieval checks whether an output is closer to its own sentence than to randomly chosen test sentences. Because the test set mixes movie reviews and biographies, a model that only identifies the category beats every distractor from the other category. FD, Self-BLEU and head entropy assess all outputs as a set, never output against target, so returning random real training sentences would score well on them; the paper does not report that baseline. The noise test feeds random values straight into the model's internal EEG representation, where nearly any model would produce gibberish, and it never checks whether swapping in another sentence's EEG changes the output to match that sentence. Finally, SemKey still invents names and facts on real EEG (Table m). The claim to have escaped the BLEU Trap is therefore not established.

Questions for authors

Q1. How accurate are the four hints when they are predicted without the brain signal, from only the session information, only the recording length, or scrambled EEG? How do they compare with naive baseline, e.g., always guessing the most common label?

Q2. How do SemKey and GLIM score when retrieval distractors come from the same category (reviews against reviews, biographies against biographies)? How does a baseline that returns random real training sentences from the right category score on your metrics?

Q3. If you feed in the EEG recorded for a different sentence, does the output change to match that sentence?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.