acceptodds
Under review as a conference paper at ICLR 2027

Instruct, Reason, Decode: Fine-Grained Semantic Instruction and Reasoning for EEG-to-Text Decoding

Abstract

Decoding open-vocabulary text from electroencephalography (EEG) remains challenging because of the low signal-to-noise ratio of EEG and the complexity of natural language. Although large language models can generate coherent text, limited understanding of EEG semantics can lead to excessive reliance on linguistic priors. During autoregressive generation, early prediction errors can also affect subsequent predictions. Existing approaches improve EEG–text alignment or introduce auxiliary semantic tasks, but their sentence-level learning objectives and coarse-grained prompts provide limited guidance about the specific content to express. To address these challenges, we propose SIRE (Semantic Instruction and Reasoning from EEG), an EEG-to-text framework that strengthens fine-grained semantic understanding and provides semantic guidance for generation. Task-specific question–answer instruction provides explicit learning targets for decoding evaluation aspects, judgments, factual categories, and semantic relations from EEG. Reasoning from EEG then adaptively selects semantic questions and integrates the predicted answers through chain-of-thought reasoning before sentence generation. These answers remain available throughout autoregressive decoding, allowing the model to use the predicted semantics alongside EEG representations and previously generated text. Experiments on ZuCo 1.0 and ZuCo 2.0 show that SIRE outperforms the compared baselines across multiple semantic decoding metrics under fully autoregressive generation. Ablations support the contributions of semantic instruction and reasoning, while input controls show that generation benefits from correctly paired EEG.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Brain–Computer Interfaces, EEG-to-Text Decoding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PKvMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SIRE, a framework for open-vocabulary EEG-to-text decoding that combines fine-grained semantic instruction tuning with intermediate QA generation. It aligns EEG representations with a Flan-T5-large language backbone and introduces task-specific QA supervision covering evaluative judgments, factual information, and semantic relations. During inference, the model selects questions and predicts their answers from EEG, then uses the resulting QA sequence to guide autoregressive sentence generation. Experiments on the combined ZuCo 1.0 and ZuCo 2.0 datasets report improvements over existing baselines on semantic matching and several text-generation metrics.

Critical strengths

The paper targets excessive reliance on language priors and investigates whether explicit semantic supervision can improve the recovery of sentence-specific content from EEG.

Critical weaknesses
  1. Limited methodological novelty and insufficient evidence for the distinctive value of the QA design. Compared with existing semantic auxiliary supervision and semantic prompting approaches, the main distinction is the construction of finer-grained, task-specific QA supervision and the organization of predicted semantic answers into an intermediate sequence before sentence generation. The contribution therefore appears incremental. Table 4 does provide evidence that QA supervision helps: without CoT, fine-grained QA improves 24-way accuracy from 11.07% to 14.11%. However, this comparison does not fully distinguish the benefits of additional semantic labels, their QA formulation, and the particular semantic targets selected. Controls using the same labels through auxiliary classification or direct semantic prompting, with training budgets matched where possible, would help establish whether the proposed QA formulation offers benefits beyond semantic auxiliary supervision.

  2. The contribution of EEG information to the performance gains requires clearer quantification. EEG-to-Text performs near chance on semantic matching in Table 2, yet achieves a BLEU-1 score of 20.28 in Table 3. These results are not contradictory; they illustrate that lexical overlap can remain substantial without effective sentence-level semantic discrimination. BLEU alone therefore cannot establish recovery of target content from EEG. The shuffled-EEG and noise controls in Figure 2 support dependence on correctly paired EEG, but do not establish how much of the improvement from QA supervision reflects better use of EEG information. The same input controls should be applied to the no-QA, coarse-QA, and fine-QA variants, alongside a no-EEG baseline with matched model capacity and training budget. Reporting the gains from real EEG over these controls would strengthen the attribution. Furthermore, the exclusive use of Flan-T5-large leaves the dependence of the findings on this particular language backbone unresolved.

  3. The evaluation is too narrow to establish the generalizability of the QA and semantic-target design. ZuCo 1.0 and ZuCo 2.0 are combined before sentence-level train/validation/test splitting, rather than providing independent cross-domain validation. Moreover, the six questions and their answer spaces are specifically designed around the reading tasks in these datasets. The current results therefore do not adequately distinguish transferable decoding improvements from adaptation to a particular task and corpus. Evaluation on an independent EEG–text dataset, or a rigorous cross-dataset or cross-task transfer setting, would help determine whether the same QA design remains effective. The paper should also report the number of unique sentences, rather than only EEG–text pairs, and assess the stability of the improvements using multiple random seeds or confidence intervals.

  4. The mathematical definitions and implementation details need greater precision. For example, Eq. (10) does not specify the exact normalization operation or its dimensions. The EEG representation contains 96 fixed latent positions, but the construction of the corresponding text sequence representation—and the length and positional matching required for element-wise MSE—remains unclear. These operations should be defined explicitly to clarify the alignment objective and support reproducibility.

Questions for authors

See the weakness.

7 comments
AuthorsOpenReview ↗

Thanks for the careful evaluation of our work and for the valuable comments and constructive suggestions. We appreciate your recognition of the motivation of SIRE and your constructive feedback regarding methodological novelty, EEG dependence, generalizability, and reproducibility. Please find our responses below.

[W1] Limited methodological novelty and insufficient evidence for the distinctive value of QA formulation.

Thank you for this valuable comment. To address this concern, we implemented an auxiliary classification baseline using the same Q1–Q6 semantic labels as SIRE, with auxiliary classification heads providing fine-grained semantic supervision without explicit QA-based reasoning. Both approaches share the same Stage 1 initialization and matched training configurations to ensure a fair comparison.

The results of this additional experiment are presented below. All metrics are reported as percentages (%), with higher values indicating better performance.

Method2-way ↑4-way ↑10-way ↑24-way ↑BLEU-1 ↑BLEU-2 ↑BLEU-3 ↑BLEU-4 ↑
Auxiliary Classification70.9848.8028.1013.8524.207.922.890.89
SIRE (Ours)73.3850.5028.3614.8526.8510.163.962.18

SIRE outperforms the auxiliary classification baseline across all eight metrics. Since both methods employ the same fine-grained semantic labels, these results suggest that SIRE's improvements are not solely attributable to additional semantic supervision, but also benefit from its explicit QA-based reasoning formulation. This provides further empirical support for the distinctive contribution of structured semantic inquiry and reasoning to EEG-to-text decoding.

We have added this comparison to Tables 4 and 5.

[W2.1] The same input controls should be applied to the no-QA, coarse-QA, and fine-QA variants.

Thank you for this valuable suggestion. We agree that consistent EEG input controls across different QA configurations are important for evaluating the contribution of fine-grained semantic supervision.

To address this concern, we applied three input conditions to the no-QA, coarse-QA, and fine-QA configurations during both training and testing: (1) Real EEG, using the original preprocessed signals; (2) Shuffled EEG, randomly reassigning EEG samples to mismatched sentences to disrupt EEG–text correspondence; and (3) Noise Control (Phase-Randomized EEG), independently randomizing Fourier phases across EEG channels while preserving their amplitude spectra, followed by the same normalization as real EEG. All configurations follow the same evaluation protocol. We report 2-way and 24-way semantic matching accuracy (%), with higher values indicating better performance.

Table 1. Real EEG

QA Configuration2-way (%) ↑24-way (%) ↑
No QA68.7911.07
Coarse QA71.0612.71
Fine QA (SIRE)73.3814.85

Table 2. Shuffled EEG

QA Configuration2-way (%) ↑24-way (%) ↑
No QA65.538.03
Coarse QA65.128.77
Fine QA (SIRE)66.339.13

Table 3. Noise Control (Phase-Randomized EEG)

QA Configuration2-way (%) ↑24-way (%) ↑
No QA58.085.33
Coarse QA54.665.66
Fine QA (SIRE)54.475.12

The results show that finer-grained QA supervision consistently improves performance with real EEG, while the gains are substantially reduced under shuffled EEG and noise controls. In particular, the real–shuffled gap increases from 3.04 to 5.72 percentage points in 24-way accuracy, whereas phase randomization substantially degrades SIRE's performance. These findings suggest that fine-grained QA supervision enhances the utilization of meaningful EEG information rather than merely strengthening language priors.

[W2.2] The exclusive use of Flan-T5-large leaves the dependence of the findings on this particular language backbone unresolved

Thank you for this valuable observation. We agree that evaluating SIRE with different language backbones is important for assessing its generalizability.

To address this concern, we extended SIRE to BART-Large and Qwen2.5-1.5B, covering both encoder–decoder and decoder-only architectures. All experiments follow the same data splits and evaluation protocol, while retaining fine-grained QA supervision and reasoning-guided generation, with necessary architecture-specific adaptations. The table reports N-way semantic matching accuracy and BLEU scores (%), with higher values indicating better performance.

Language Backbone2-way ↑4-way ↑10-way ↑24-way ↑BLEU-1 ↑BLEU-2 ↑BLEU-3 ↑BLEU-4 ↑
BART-Large64.7538.6117.997.8925.318.592.721.31
Qwen2.5-1.5B71.3448.3726.3313.1927.9811.504.402.11
Flan-T5-large (Ours)73.3850.5028.3614.8526.8510.163.962.18

The results demonstrate that SIRE can be implemented across both encoder–decoder and decoder-only architectures. Flan-T5-large achieves the best semantic matching performance, while Qwen2.5-1.5B achieves competitive N-way accuracy and even higher BLEU-1, BLEU-2, and BLEU-3 scores. These findings suggest that different language backbones exhibit complementary strengths in semantic matching and lexical generation, supporting the architectural flexibility of SIRE beyond Flan-T5-large.

We have added this comparison to Table 18 in Appendix B.8.

[W3.1] The evaluation is too narrow to establish the generalizability of the QA and semantic-target design.

Thank you for highlighting this important limitation. We acknowledge that our current evaluation is restricted to ZuCo 1.0 and ZuCo 2.0, which does not fully establish the cross-dataset generalizability of SIRE.

Cross-dataset EEG-to-text decoding remains challenging due to differences in recording protocols, inter-subject variability, and dataset-specific semantic structures. Potential solutions include cross-dataset EEG representation alignment, designing diverse and task-adaptive QA schemas to accommodate different linguistic tasks, and exploring implicit chain-of-thought reasoning to reduce reliance on predefined QA structures.

We are currently extending our evaluation to additional EEG-to-text datasets with different recording protocols and linguistic tasks. We will clarify this limitation and discuss these potential directions in the revised manuscript.

We have added this discussion to Section 5.

[W3.2] Unique sentence statistics and statistical reliability.

Thank you for this valuable suggestion. We agree that reporting unique sentence counts and variability across random seeds is important for ensuring reliable evaluation.

We now report the numbers of EEG–text pairs and unique sentences in each data split. We also repeated the experiments using four different random seeds and report the mean and sample standard deviation across the four runs. This analysis covers semantic matching, content recovery, distributional alignment, lexical overlap, and generation diversity.

Table 1: Dataset statistics.

Data SplitEEG–Text PairsUnique Sentences
Training10,9991,066
Validation1,672153
Test3,206305
Total15,8771,524

Table 2: SIRE performance across four random seeds.

Statistic2-way ↑4-way ↑10-way ↑24-way ↑C. Recall ↑FD ↓BLEU-1 ↑BLEU-2 ↑BLEU-3 ↑BLEU-4 ↑Dist-1 ↑Dist-2 ↑Self-BLEU ↓H. Ent. ↑
Mean ± Std73.12 ± 0.2950.25 ± 0.3428.33 ± 0.4014.92 ± 0.310.0307 ± 0.00170.4174 ± 0.024125.81 ± 1.089.44 ± 0.653.18 ± 0.551.39 ± 0.550.0376 ± 0.00250.1285 ± 0.006276.20 ± 4.454.14 ± 0.66

The held-out test set contains 3,206 EEG–text pairs corresponding to 305 unique sentences. Across four independent training runs, SIRE achieves stable semantic matching performance, with standard deviations of 0.29–0.40 percentage points and a mean 24-way accuracy of 14.92 ± 0.31%, consistent with the originally reported 14.85%. These results support the reproducibility of SIRE's main findings while providing greater transparency regarding dataset statistics.

We have updated Tables 2 and 3 with these results.

[W4] The mathematical definitions and implementation details need greater precision.

Thank you for this comment. We would like to clarify that the normalization operation and its dimensions are already explicitly defined in Appendix A.1 (EEG Representation Alignment). Specifically, Norm denotes L2 normalization along the last feature dimension ((d=1024)), applied independently at each of the (L=96) sequence positions, without normalization across positions or batch samples. Equation (10) computes the element-wise MSE averaged over the batch, sequence, and feature dimensions.

Furthermore, the positional correspondence represents a learned alignment objective between EEG and contextualized text representations, rather than a predefined one-to-one mapping between EEG time segments and text tokens.

Reviewer_8gDLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SIRE for open-vocabulary EEG-to-text decoding. Instead of relying only on sentence-level EEG–text alignment, SIRE introduces task-specific fine-grained question–answer supervision to teach the language model semantic distinctions such as evaluation aspects, factual categories, predicates, and semantic relations. At inference, the model adaptively selects and answers semantic questions from EEG, forms a QA-based reasoning history, and conditions autoregressive sentence generation on both the EEG representations and the predicted semantic answers. Experiments on ZuCo 1.0 and ZuCo 2.0 show improved semantic matching and content recovery over several EEG-to-text baselines, while ablations support the contributions of fine-grained QA supervision, CoT guidance, and LoRA adaptation. Shuffled-EEG and Gaussian-noise controls further indicate that the generated outputs benefit from correctly paired EEG signals.

Critical strengths
  1. The method directly targets a plausible weakness of current EEG-to-text systems. Rather than relying only on sentence-level alignment, SIRE introduces explicit fine-grained semantic supervision before generation.

  2. The ablation study is relatively convincing. Fine-grained QA alone produces a large improvement, CoT adds further gains, and LoRA contributes additional performance, so the main components are reasonably well separated.

  3. The EEG-dependency analysis is appropriate. Shuffled-EEG and Gaussian-noise controls show that the gains depend on correctly paired EEG rather than only linguistic priors.

Critical weaknesses
  1. Most of the gain appears to come from fine-grained semantic supervision rather than the reasoning component itself. In the ablation, fine-grained QA without CoT already improves 24-way accuracy from 11.07% to 14.11%, while adding CoT further increases it to 14.85%, suggesting that semantic instruction is the dominant contributor.

  2. The paper does not directly evaluate the accuracy of the intermediate semantic answers. Since the central mechanism relies on predicted QA responses to guide generation, reporting question-wise semantic prediction accuracy would help verify whether the improvement comes from correctly decoded EEG semantics rather than from the additional structured generation context.

  3. The semantic supervision is highly task-specific. The six QA types are manually designed around the three ZuCo reading tasks, with different semantic schemas for sentiment, factual content, and relation reading, so it remains unclear how easily the framework transfers to datasets with different linguistic tasks or annotation structures.

  4. The evaluation is restricted to the ZuCo family. Although ZuCo 1.0 and 2.0 cover multiple reading tasks, they share a similar experimental setting, so the transferability of the QA-based semantic guidance to substantially different EEG-to-text paradigms remains untested.

Questions for authors
  1. How are errors distributed across the six semantic questions?

  2. Why does adaptive QA selection improve semantic matching but not consistently improve lexical-overlap metrics over fixed-order QA?

  3. How sensitive is SIRE to the specific QA schema and answer-space design?

4 comments
AuthorsOpenReview ↗

Thanks for the positive assessment of our work and for the valuable comments and constructive suggestions. We appreciate your recognition of the motivation, ablation studies, and EEG-dependency analysis of SIRE. Please find our responses below.

[W1] Contribution of CoT Reasoning

Thank you for this insightful observation. We agree that fine-grained QA supervision plays an important role in improving semantic recovery. However, we would like to emphasize that CoT provides additional benefits beyond fine-grained semantic supervision, as evidenced by improvements in semantic matching, content recovery, and language generation.

We compare three configurations under fully autoregressive generation on ZuCo 1.0 and ZuCo 2.0, progressively introducing Fine QA supervision and CoT while keeping LoRA enabled.

QA InstructionCoT2-way ↑4-way ↑10-way ↑24-way ↑C. Recall ↑Fréchet ↓
None✗68.7944.3523.0611.070.02330.5924
Fine✗71.0648.3527.0914.110.02730.4528
Fine✓73.3850.5028.3614.850.03260.3951

Under the same Fine QA supervision, incorporating CoT consistently improves all four semantic matching accuracies and content-word recall while reducing Fréchet distance. Although the incremental gain in 24-way accuracy is relatively modest, the consistent improvements across these metrics support the additional contribution of CoT beyond fine-grained semantic supervision alone.

To further assess the contribution of CoT to language generation, we evaluate the same three configurations using BLEU-1 through BLEU-4 (%), with higher values indicating better lexical overlap. All other experimental settings remain unchanged.

QA InstructionCoTBLEU-1 ↑BLEU-2 ↑BLEU-3 ↑BLEU-4 ↑
None✗26.709.763.181.49
Fine✗24.988.422.351.04
Fine✓26.8510.163.962.18

Notably, introducing Fine QA without CoT improves semantic matching but reduces BLEU scores relative to the no-QA baseline. Incorporating CoT reverses this decline, surpassing the no-QA baseline across all four BLEU metrics, with particularly substantial improvements of 68.5% in BLEU-3 and 109.6% in BLEU-4 over Fine QA without CoT. These findings highlight the complementary roles of Fine QA and CoT: while Fine QA strengthens semantic supervision, CoT helps translate intermediate semantic information into more effective language generation, providing additional benefits across semantic matching, content recovery, and lexical-overlap metrics.

We have added these results to Tables 4 and 5.

[W2, Q1] Intermediate Semantic Prediction Accuracy

Thank you for this valuable observation. We agree that improvements in final sentence-matching performance do not directly establish the accuracy of intermediate semantic predictions.

To address this concern, we evaluated all six semantic targets on the held-out test set by comparing model-predicted answers against their reference annotations. The table reports prediction accuracy (%) for each semantic target, with higher values indicating better performance.

Semantic TargetAspectJudgmentFact typeFactual predicateRelation typeRelational predicate
Accuracy (%) ↑83.5753.2382.1283.5775.6271.25

To further examine Judgment prediction, we additionally evaluated three sentiment categories by merging Very Positive with Positive and Very Negative with Negative. The table reports class-wise accuracy (%), with higher values indicating better performance.

Judgment (3-class)PositiveNegativeNeutral
Accuracy (%) ↑78.9679.0374.47

The results demonstrate that SIRE achieves strong prediction accuracy across several intermediate semantic targets, particularly Aspect, Fact type, and Factual predicate. The improved three-class Judgment results further suggest that SIRE captures broader sentiment polarity more effectively than subtle differences in sentiment intensity. Overall, these findings provide direct quantitative evidence for the recovery of meaningful intermediate semantic information, supporting **the effectiveness of fine-grained QA supervision **in SIRE.

We have added these results to Tables 14 and 15 in Appendix B.4.

[W3, Q3] Task-Specific Semantic Supervision

Thanks for this valuable observation. We acknowledge that the current QA schema is specifically designed for the three ZuCo reading tasks, which limits its direct generalizability to other datasets and linguistic settings.

However, the core idea of SIRE is not tied to these specific questions, but rather to fine-grained semantic supervision and reasoning-guided generation. Potential directions include automatically generating task-adaptive QA schemas, developing unified semantic representations across tasks, and exploring implicit chain-of-thought reasoning to reduce reliance on predefined questions and answer categories while preserving semantic guidance.

We have added this discussion to Section 5.

[W4] Cross-Dataset Generalization

Thanks for highlighting this important limitation. We acknowledge that our current evaluation is restricted to ZuCo 1.0 and ZuCo 2.0, which share similar experimental settings and therefore do not fully establish the generalizability of SIRE to different EEG-to-text paradigms.

We are currently working on extending our evaluation to additional EEG-to-text datasets with different recording protocols and linguistic tasks. We also plan to investigate cross-dataset transfer learning and task-adaptive QA schema construction to improve the generalizability of SIRE across different experimental settings.

We have clarified this limitation in Section 5.

[Q2] Adaptive vs. Fixed-Order QA

Thank you for this insightful question. As highlighted by SemKey (Wang et al., 2026), BLEU scores in EEG-to-text decoding may be influenced by repetitive sentence templates and frequently occurring phrases, without necessarily indicating better semantic recovery.

To investigate this issue, we compare fixed-order and adaptive QA generation under the same Fine QA supervision and LoRA settings.

  • Self-BLEU: Measures similarity among generated sentences, where higher values indicate greater repetition and lower generation diversity.

The table reports Self-BLEU (%), with lower values indicating better generation diversity.

QA PolicySelf-BLEU (%) ↓
Fixed order90.00
Adaptive (SIRE)75.16

The substantially lower Self-BLEU of adaptive QA (75.16% vs. 90.00%) indicates reduced repetition, suggesting that the slightly higher BLEU-1 and BLEU-2 scores of fixed-order QA may partly reflect repetitive lexical patterns rather than better semantic recovery.

We have added this comparison to Tables 21 and 22 in Appendix B.10.

Reviewer_8gDLOpenReview ↗

Official Comment

Thank you for the detailed response and clarifications. I will maintain my original rating.

Thank you for your insightful comments and your efforts in reviewing our manuscript. We hope our responses have addressed your concerns. Thank you!

Reviewer_FYeAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SIRE studies EEG-to-text decoding during reading, aiming to provide more specific semantic guidance before sentence generation and reduce reliance on language priors. An initial stage aligns EEG and text through contrastive learning, representation regression, and sentence reconstruction. Instruction tuning then freezes the EEG network and Flan-T5-large base weights while updating the projector, LoRA, and structural tokens. Six QA fields cover evaluation aspects and judgments, factual categories and predicates, and relation types and associated predicates across three ZuCo reading tasks. Qwen2.5-7B annotates the first four fields from training and validation sentences, with separate verification requests and rule checks. Relation labels come from the dataset, and Q6 is a fixed mapping from Q5. At inference, the model selects unused questions from the complete bank, predicts answers, and decides when to stop before generating a sentence. QA history and EEG representations remain accessible throughout decoding. The complete output shares a 192-token limit, with no ground-truth QA or reference-text prefix.

Experiments combine ZuCo 1.0 and 2.0 with a 70/10/20 split by sentence identity, keeping repeated recordings of a sentence together. After preprocessing, there are 10,999 training, 1,672 validation, and 3,206 test EEG observations. Comparisons use semantic matching, content-word recall, distributional distance, and lexical metrics. The 24-way matching score rises from SemKey’s 11.02% to 14.85%, and content-word recall from 0.0168 to 0.0326. Ablations obtain 11.07% without QA, 14.11% with fine QA but no CoT prefix, and 14.85% for full SIRE. Coarse QA, removal of LoRA, and fixed-order QA provide further controls. Input tests exchange different-sentence EEG within dataset, participant, and reading-task groups, regenerating QA and sentences; correctly paired EEG performs better.

Critical strengths
  1. Recordings of the same sentence do not cross training and test partitions, evaluation is fully autoregressive, and no true semantic answers are supplied. The shuffle excludes the same sentence within dataset, participant, and reading-task groups, moves the EEG mask with the recording, and regenerates intermediate QA. This tests input correspondence more directly than noise replacement alone, and the results support dependence on the target EEG. Empty predictions and malformed outputs remain in evaluation.

  2. Table 4 holds the EEG encoder, language backbone, split, and evaluation fixed. Fine QA increases 24-way matching from 11.07% to 14.11% without a reasoning prefix; full SIRE reaches 14.85%. This supports a useful role for finer auxiliary supervision. The appendix also compares fixed and adaptive QA, obtaining 12.89% and 14.85%, while acknowledging that selection and termination change along with ordering.

Critical weaknesses
  1. The paper does not directly report correctness or confusion for the six QA fields on unseen sentences, independent validation of their annotations, or how wrong QA changes the final sentence. The first four fields use the same model for candidate generation and verification, and agent-assisted review is explicitly not an independent human annotation study. Q6 is a fixed mapping from Q5 rather than an independently recovered semantic distinction. Better final-sentence matching supports the auxiliary training as a whole, but does not establish which fine distinctions are decoded or directly validate the proposed reduction of autoregressive error propagation.

  2. The fine-QA CoT increment in Table 4 is 0.74 percentage points in 24-way matching; adaptive versus fixed QA adds 1.96 points in the appendix. Neither has independent training repetitions or an interval for the difference. The implementation lists alignment seed 42 and instruction-tuning seed 648, which are not repeated runs of each condition. The 3,206 test observations include repeated sentence recordings, and 10,000 candidate draws only rescore fixed outputs. Since fine QA and the reasoning policy carry the main increment, stability directly matters to the novelty assessment. Observation-based candidate sampling without reference deduplication also leaves unclear how much improvement concerns fine distinctions within a reading task.

Questions for authors
  1. Could an independently annotated subset of unseen sentences provide per-field QA accuracy and confusion, with examples linking predicted QA to the final sentence? With other inputs fixed, how do equally long incorrect or uninformative QA prefixes affect gains and subsequent errors?

  2. Do the fine-QA CoT and adaptive-versus-fixed QA differences persist across repeated training? Could evaluation uncertainty be clustered by unique sentence, while separating training variation from candidate-sampling variation?

  3. How do relative gains change with deduplicated references and semantically close candidates from the same reading task? This would clarify fine semantic improvement versus broader category discrimination.

5 comments
AuthorsOpenReview ↗

We sincerely thank you for your valuable comments and constructive suggestions. Please find our responses below.

[W1.1] Reliability of QA Annotations

Thank you for raising concerns about the reliability and independent validation of QA annotations. Our original annotation pipeline includes candidate generation, verification, and rule-based consistency checks. Given the substantial effort required for comprehensive manual verification, we conducted an additional cross-model audit using three independent language models outside the original Qwen annotation family: GPT-OSS-20B, Mistral-Small-3.2-24B, and Gemma-4-31B. Each model independently evaluated the original annotations for Aspect, Judgment, Fact type, and Factual predicate.

To further investigate the relatively low five-class Judgment agreement, we additionally evaluated agreement after merging adjacent sentiment intensities into three categories: Positive (Positive/Very Positive), Neutral (Mixed or Neutral), and Negative (Negative/Very Negative).

QA Annotation Agreement:

Audit ModelAspect Agreement (%) ↑Judgment Agreement (5-class, %) ↑Judgment Agreement (3-class, %) ↑Fact type Agreement (%) ↑Factual predicate Agreement (%) ↑
GPT-OSS-20B81.2042.4090.1668.0069.60
Mistral-Small-3.2-24B75.4053.6086.9252.9078.70
Gemma-4-31B77.9044.0083.9466.1067.00

The results demonstrate consistent cross-model agreement on several semantic attributes, providing additional support for the reliability of our QA annotations. Importantly, the substantial improvement in Judgment agreement after merging adjacent intensity levels suggests that disagreements are primarily associated with fine-grained sentiment distinctions rather than overall sentiment polarity. These findings support the consistency of the underlying semantic annotations while highlighting the ambiguity of fine-grained sentiment classification.

We have added these results to Tables 9 and 10 in Appendix A.2.

[W1.2, Q1] Accuracy of Intermediate Semantic Predictions

Thank you for highlighting the importance of evaluating intermediate semantic predictions. We agree that improvements in final sentence-matching performance alone do not directly establish the accuracy of individual QA fields.

To address this concern, we evaluated the prediction accuracy of all six semantic targets on the held-out test set by comparing predicted answers against their reference annotations. The table reports accuracy (%), with higher values indicating better performance.

Semantic TargetAspectJudgment (5-class)Fact typeFactual predicateRelation typeRelational predicate
Accuracy (%) ↑83.5753.2382.1283.5775.6271.25

To further examine Judgment prediction, we additionally evaluated three sentiment categories by merging Very Positive with Positive and Very Negative with Negative, while retaining Mixed or Neutral as Neutral. The table reports class-wise accuracy (%), with higher values indicating better performance.

Judgment (3-class)PositiveNegativeNeutral
Accuracy (%) ↑78.9679.0374.47

The results show that Aspect, Fact type, and Factual predicate achieve over 80% accuracy. Although five-class Judgment prediction is more challenging, the improved three-class results suggest that SIRE can recover broader sentiment information more effectively than subtle distinctions in sentiment intensity.

We further provide a representative example illustrating QA-guided reasoning and subsequent sentence generation.

Example: Sentiment Semantic Recovery

Predicted QA reasoning and generated sentence:

<COT>:
  Question: What aspect of the movie is being evaluated?
  Predicted Answer: OVERALL_QUALITY

  Question: What fine-grained judgment is expressed?
  Predicted Answer: VERY_NEGATIVE
<FINAL>
  sluggish, sloppy mess.
</FINAL>

Ground Truth (GT):

... pitiful, slapdash disaster.

This example illustrates that intermediate QA predictions can preserve salient sentiment information despite lexical differences from the reference. Together with the improved three-class Judgment results, it supports SIRE's ability to recover meaningful semantic attributes. Moreover, Fine QA supervision improves 24-way semantic matching accuracy from 11.07% to 14.11% (+27.5% relative)*without CoT, while Full SIRE further achieves 14.85%. Overall, these findings support the effectiveness of fine-grained QA supervision in providing structured semantic information for EEG-conditioned language generation.

We have added these results to Tables 14 and 15 in Appendix B.4.

[W1.3, Q1] Effects of QA Prefixes on Final Generation

Thank you for raising the question of how intermediate QA predictions affect final sentence generation. We agree that directly examining the influence of QA content is important for understanding the role of intermediate semantic reasoning in SIRE.

To investigate this, we conducted controlled QA-prefix intervention experiments while keeping the EEG inputs, model parameters, and decoding settings fixed. Specifically, after the model selected the QA fields and generated their answers, we replaced the predicted answers with three types of QA prefixes: (1) Correct QA, using the corresponding reference answers; (2) Incorrect QA, replacing the answers with incorrect labels; and (3) Uninformative QA, replacing the answers with non-informative labels. The selected questions and prefix structure were preserved, isolating the effects of QA answer content on final generation. We evaluated 2-way and 24-way semantic matching accuracy (%), with higher values indicating better performance.

QA Prefix2-way Accuracy (%) ↑24-way Accuracy (%) ↑
Correct QA74.7714.92
Incorrect QA62.439.63
Uninformative QA60.577.67

The results show that correct QA prefixes consistently outperform incorrect and uninformative ones, demonstrating that intermediate semantic information meaningfully guides final generation rather than merely adding prefix tokens. These findings support the effectiveness of SIRE's QA-guided reasoning mechanism while highlighting the importance of accurate intermediate predictions.

We have added this experiment to Table 16 in Appendix B.5.

[W2.1, Q2] Stability Across Multiple Training Seeds

Thank you for highlighting the importance of statistical robustness. We agree that evaluating performance across independent training runs is essential for assessing the reliability and reproducibility of the reported improvements.

To address this concern, we conducted four independent training runs with different random seeds. All runs followed the same dataset split, model architecture, training hyperparameters, and evaluation protocol, with only the random seed varied. We report the mean and sample standard deviation across the four runs for semantic matching accuracy (2-way, 4-way, 10-way, and 24-way), content-word recall, Fréchet distance, BLEU-1 through BLEU-4, and generation diversity metrics (Distinct-1/2, Self-BLEU, and head entropy).

Statistic2-way ↑4-way ↑10-way ↑24-way ↑C. Recall ↑FD ↓BLEU-1 ↑BLEU-2 ↑BLEU-3 ↑BLEU-4 ↑Dist-1 ↑Dist-2 ↑Self-BLEU ↓H. Ent. ↑
Mean ± Std73.12 ± 0.2950.25 ± 0.3428.33 ± 0.4014.92 ± 0.310.0307 ± 0.00170.4174 ± 0.024125.81 ± 1.089.44 ± 0.653.18 ± 0.551.39 ± 0.550.0376 ± 0.00250.1285 ± 0.006276.20 ± 4.454.14 ± 0.66

N-way accuracies, BLEU, and Self-BLEU are reported as percentages; content-word recall and Distinct-1/2 as ratios; head entropy in bits; and Fréchet distance as an unscaled value. Higher values indicate better performance, except for Fréchet distance and Self-BLEU.

The results demonstrate that SIRE maintains stable semantic decoding performance across different random seeds, with consistently low variability in semantic matching accuracy. The close agreement with the originally reported results further supports the reliability and reproducibility of SIRE's performance. We have updated Tables 2 and 3 with these results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.