- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SIRE, a framework for open-vocabulary EEG-to-text decoding that combines fine-grained semantic instruction tuning with intermediate QA generation. It aligns EEG representations with a Flan-T5-large language backbone and introduces task-specific QA supervision covering evaluative judgments, factual information, and semantic relations. During inference, the model selects questions and predicts their answers from EEG, then uses the resulting QA sequence to guide autoregressive sentence generation. Experiments on the combined ZuCo 1.0 and ZuCo 2.0 datasets report improvements over existing baselines on semantic matching and several text-generation metrics.
Critical strengths
The paper targets excessive reliance on language priors and investigates whether explicit semantic supervision can improve the recovery of sentence-specific content from EEG.
Critical weaknesses
-
Limited methodological novelty and insufficient evidence for the distinctive value of the QA design. Compared with existing semantic auxiliary supervision and semantic prompting approaches, the main distinction is the construction of finer-grained, task-specific QA supervision and the organization of predicted semantic answers into an intermediate sequence before sentence generation. The contribution therefore appears incremental. Table 4 does provide evidence that QA supervision helps: without CoT, fine-grained QA improves 24-way accuracy from 11.07% to 14.11%. However, this comparison does not fully distinguish the benefits of additional semantic labels, their QA formulation, and the particular semantic targets selected. Controls using the same labels through auxiliary classification or direct semantic prompting, with training budgets matched where possible, would help establish whether the proposed QA formulation offers benefits beyond semantic auxiliary supervision.
-
The contribution of EEG information to the performance gains requires clearer quantification. EEG-to-Text performs near chance on semantic matching in Table 2, yet achieves a BLEU-1 score of 20.28 in Table 3. These results are not contradictory; they illustrate that lexical overlap can remain substantial without effective sentence-level semantic discrimination. BLEU alone therefore cannot establish recovery of target content from EEG. The shuffled-EEG and noise controls in Figure 2 support dependence on correctly paired EEG, but do not establish how much of the improvement from QA supervision reflects better use of EEG information. The same input controls should be applied to the no-QA, coarse-QA, and fine-QA variants, alongside a no-EEG baseline with matched model capacity and training budget. Reporting the gains from real EEG over these controls would strengthen the attribution. Furthermore, the exclusive use of Flan-T5-large leaves the dependence of the findings on this particular language backbone unresolved.
-
The evaluation is too narrow to establish the generalizability of the QA and semantic-target design. ZuCo 1.0 and ZuCo 2.0 are combined before sentence-level train/validation/test splitting, rather than providing independent cross-domain validation. Moreover, the six questions and their answer spaces are specifically designed around the reading tasks in these datasets. The current results therefore do not adequately distinguish transferable decoding improvements from adaptation to a particular task and corpus. Evaluation on an independent EEG–text dataset, or a rigorous cross-dataset or cross-task transfer setting, would help determine whether the same QA design remains effective. The paper should also report the number of unique sentences, rather than only EEG–text pairs, and assess the stability of the improvements using multiple random seeds or confidence intervals.
-
The mathematical definitions and implementation details need greater precision. For example, Eq. (10) does not specify the exact normalization operation or its dimensions. The EEG representation contains 96 fixed latent positions, but the construction of the corresponding text sequence representation—and the length and positional matching required for element-wise MSE—remains unclear. These operations should be defined explicitly to clarify the alignment objective and support reproducibility.
Questions for authors
See the weakness.
7 comments
[W2.2] The exclusive use of Flan-T5-large leaves the dependence of the findings on this particular language backbone unresolved
Thank you for this valuable observation. We agree that evaluating SIRE with different language backbones is important for assessing its generalizability.
To address this concern, we extended SIRE to BART-Large and Qwen2.5-1.5B, covering both encoder–decoder and decoder-only architectures. All experiments follow the same data splits and evaluation protocol, while retaining fine-grained QA supervision and reasoning-guided generation, with necessary architecture-specific adaptations. The table reports N-way semantic matching accuracy and BLEU scores (%), with higher values indicating better performance.
| Language Backbone | 2-way ↑ | 4-way ↑ | 10-way ↑ | 24-way ↑ | BLEU-1 ↑ | BLEU-2 ↑ | BLEU-3 ↑ | BLEU-4 ↑ |
|---|---|---|---|---|---|---|---|---|
| BART-Large | 64.75 | 38.61 | 17.99 | 7.89 | 25.31 | 8.59 | 2.72 | 1.31 |
| Qwen2.5-1.5B | 71.34 | 48.37 | 26.33 | 13.19 | 27.98 | 11.50 | 4.40 | 2.11 |
| Flan-T5-large (Ours) | 73.38 | 50.50 | 28.36 | 14.85 | 26.85 | 10.16 | 3.96 | 2.18 |
The results demonstrate that SIRE can be implemented across both encoder–decoder and decoder-only architectures. Flan-T5-large achieves the best semantic matching performance, while Qwen2.5-1.5B achieves competitive N-way accuracy and even higher BLEU-1, BLEU-2, and BLEU-3 scores. These findings suggest that different language backbones exhibit complementary strengths in semantic matching and lexical generation, supporting the architectural flexibility of SIRE beyond Flan-T5-large.
We have added this comparison to Table 18 in Appendix B.8.
[W3.1] The evaluation is too narrow to establish the generalizability of the QA and semantic-target design.
Thank you for highlighting this important limitation. We acknowledge that our current evaluation is restricted to ZuCo 1.0 and ZuCo 2.0, which does not fully establish the cross-dataset generalizability of SIRE.
Cross-dataset EEG-to-text decoding remains challenging due to differences in recording protocols, inter-subject variability, and dataset-specific semantic structures. Potential solutions include cross-dataset EEG representation alignment, designing diverse and task-adaptive QA schemas to accommodate different linguistic tasks, and exploring implicit chain-of-thought reasoning to reduce reliance on predefined QA structures.
We are currently extending our evaluation to additional EEG-to-text datasets with different recording protocols and linguistic tasks. We will clarify this limitation and discuss these potential directions in the revised manuscript.
We have added this discussion to Section 5.
[W3.2] Unique sentence statistics and statistical reliability.
Thank you for this valuable suggestion. We agree that reporting unique sentence counts and variability across random seeds is important for ensuring reliable evaluation.
We now report the numbers of EEG–text pairs and unique sentences in each data split. We also repeated the experiments using four different random seeds and report the mean and sample standard deviation across the four runs. This analysis covers semantic matching, content recovery, distributional alignment, lexical overlap, and generation diversity.
Table 1: Dataset statistics.
| Data Split | EEG–Text Pairs | Unique Sentences |
|---|---|---|
| Training | 10,999 | 1,066 |
| Validation | 1,672 | 153 |
| Test | 3,206 | 305 |
| Total | 15,877 | 1,524 |
Table 2: SIRE performance across four random seeds.
| Statistic | 2-way ↑ | 4-way ↑ | 10-way ↑ | 24-way ↑ | C. Recall ↑ | FD ↓ | BLEU-1 ↑ | BLEU-2 ↑ | BLEU-3 ↑ | BLEU-4 ↑ | Dist-1 ↑ | Dist-2 ↑ | Self-BLEU ↓ | H. Ent. ↑ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mean ± Std | 73.12 ± 0.29 | 50.25 ± 0.34 | 28.33 ± 0.40 | 14.92 ± 0.31 | 0.0307 ± 0.0017 | 0.4174 ± 0.0241 | 25.81 ± 1.08 | 9.44 ± 0.65 | 3.18 ± 0.55 | 1.39 ± 0.55 | 0.0376 ± 0.0025 | 0.1285 ± 0.0062 | 76.20 ± 4.45 | 4.14 ± 0.66 |
The held-out test set contains 3,206 EEG–text pairs corresponding to 305 unique sentences. Across four independent training runs, SIRE achieves stable semantic matching performance, with standard deviations of 0.29–0.40 percentage points and a mean 24-way accuracy of 14.92 ± 0.31%, consistent with the originally reported 14.85%. These results support the reproducibility of SIRE's main findings while providing greater transparency regarding dataset statistics.
We have updated Tables 2 and 3 with these results.
[W4] The mathematical definitions and implementation details need greater precision.
Thank you for this comment. We would like to clarify that the normalization operation and its dimensions are already explicitly defined in Appendix A.1 (EEG Representation Alignment). Specifically, Norm denotes L2 normalization along the last feature dimension ((d=1024)), applied independently at each of the (L=96) sequence positions, without normalization across positions or batch samples. Equation (10) computes the element-wise MSE averaged over the batch, sequence, and feature dimensions.
Furthermore, the positional correspondence represents a learned alignment objective between EEG and contextualized text representations, rather than a predefined one-to-one mapping between EEG time segments and text tokens.
Thanks for the careful evaluation of our work and for the valuable comments and constructive suggestions. We appreciate your recognition of the motivation of SIRE and your constructive feedback regarding methodological novelty, EEG dependence, generalizability, and reproducibility. Please find our responses below.
[W1] Limited methodological novelty and insufficient evidence for the distinctive value of QA formulation.
Thank you for this valuable comment. To address this concern, we implemented an auxiliary classification baseline using the same Q1–Q6 semantic labels as SIRE, with auxiliary classification heads providing fine-grained semantic supervision without explicit QA-based reasoning. Both approaches share the same Stage 1 initialization and matched training configurations to ensure a fair comparison.
The results of this additional experiment are presented below. All metrics are reported as percentages (%), with higher values indicating better performance.
SIRE outperforms the auxiliary classification baseline across all eight metrics. Since both methods employ the same fine-grained semantic labels, these results suggest that SIRE's improvements are not solely attributable to additional semantic supervision, but also benefit from its explicit QA-based reasoning formulation. This provides further empirical support for the distinctive contribution of structured semantic inquiry and reasoning to EEG-to-text decoding.
We have added this comparison to Tables 4 and 5.
[W2.1] The same input controls should be applied to the no-QA, coarse-QA, and fine-QA variants.
Thank you for this valuable suggestion. We agree that consistent EEG input controls across different QA configurations are important for evaluating the contribution of fine-grained semantic supervision.
To address this concern, we applied three input conditions to the no-QA, coarse-QA, and fine-QA configurations during both training and testing: (1) Real EEG, using the original preprocessed signals; (2) Shuffled EEG, randomly reassigning EEG samples to mismatched sentences to disrupt EEG–text correspondence; and (3) Noise Control (Phase-Randomized EEG), independently randomizing Fourier phases across EEG channels while preserving their amplitude spectra, followed by the same normalization as real EEG. All configurations follow the same evaluation protocol. We report 2-way and 24-way semantic matching accuracy (%), with higher values indicating better performance.
Table 1. Real EEG
Table 2. Shuffled EEG
Table 3. Noise Control (Phase-Randomized EEG)
The results show that finer-grained QA supervision consistently improves performance with real EEG, while the gains are substantially reduced under shuffled EEG and noise controls. In particular, the real–shuffled gap increases from 3.04 to 5.72 percentage points in 24-way accuracy, whereas phase randomization substantially degrades SIRE's performance. These findings suggest that fine-grained QA supervision enhances the utilization of meaningful EEG information rather than merely strengthening language priors.