- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SpectraLoom, an open-vocabulary EEG-to-text architecture operating on word-level EEG features extracted during natural reading. The model introduces three components: (1) Spectral Band Attention, which produces a word-specific softmax gate over eight EEG frequency-band slices while retaining the full 840-dimensional input; (2) a multi-scale encoder combining parallel one-dimensional convolutions with kernel sizes 3, 5, and 7 and a three-layer bidirectional LSTM; and (3) a cross-attention bridge that allows each word-aligned EEG representation to attend over the full EEG sequence before being supplied as input embeddings to BART-large.
Critical strengths
- Careful architectural specification and matched experimental implementation
The paper describes the architecture, training procedure, tensor shapes, trainable parameter subsets, decoding settings, split construction, and random seeds in substantial detail. Four baselines are retrained using the same split, epoch budget, early stopping rule, decoding configuration, and three seeds. This is consequential because EEG-to-text results are sensitive to evaluation protocol and decoding choices, and locally matched comparisons are more interpretable than comparisons copied from papers using incompatible splits.
- Appropriate attention to ablations, free generation, and stability
The paper does not rely exclusively on teacher-forced metrics. It reports free beam-search generation, output uniqueness, WER, multiple overlap metrics, component ablations, per-seed variability, and computational cost. Each proposed component is replaced with an alternative of the same output width, and the full model is reported as more stable than the two BART-based front ends across the three selected seeds. These are useful experimental practices for a benchmark where generation can otherwise be dominated by language-model behaviour.
Critical weaknesses
- The input controls fail to establish that the model uses EEG values
The zero-input and noise-input models match or outperform the real-EEG model on several central metrics, particularly in zero-shot evaluation. On ZuCo 2.0, zero input with the mask retained exceeds real EEG in BLEU-1, BLEU-4, ROUGE-L, and output uniqueness; the model without the mask also obtains higher ROUGE-L and lower WER. Thus, the experiments do not demonstrate that the claimed performance comes from neural information rather than the pretrained language model, sequence mask, sentence length, dataset distribution, or training-set language regularities.
- The evaluation does not isolate generalisation across readers, and output degeneracy remains severe
The split groups repeated presentations of the same normalised sentence, which appropriately prevents the same sentence from appearing in multiple partitions. However, the paper does not report a reader-held-out ZuCo 1.0 evaluation. Therefore, the in-domain experiment does not establish generalisation to unseen individuals, and may permit the model to exploit participant- or session-specific regularities. The ZuCo 2.0 evaluation uses different participants and sessions, but also introduces a corpus/session/task shift, making it difficult to attribute performance to subject generalisation specifically.
Questions for authors
-
Input controls: Can the authors run the real-EEG, Gaussian-noise, zero-input, and zero-input-without-mask experiments for all three seeds and report paired differences with uncertainty? The present single-seed controls appear to show that zero input outperforms real EEG on several headline metrics, especially on ZuCo 2.0. If that result persists, what evidence remains that SpectraLoom decodes information from EEG values?
-
Correspondence controls: Can the authors evaluate a training-time and test-time permutation control in which complete EEG sequences are reassigned across sentences while preserving sequence length, participant, task, masks, and the marginal EEG distribution? Gaussian noise and zero input alter the input distribution, whereas sentence-level or word-level permutation would directly test whether the EEG-text correspondence contributes.
-
Length and mask leakage: How much of performance can be obtained from sequence length alone? Please include a mask-only or length-only baseline with learned constant embeddings, and group results by sentence length. The strong control performance suggests that the number of fixated words or valid positions may provide substantial information.
-
Reader-held-out evaluation: Does the sentence-grouped ZuCo 1.0 split share readers across training, development, and test sets? If so, can the authors report a split that holds out readers, ideally crossed with sentence grouping? This would help distinguish linguistic decoding from reader- or session-specific fitting.
-
Statistical support: Are the improvements over T5-large or other baselines statistically reliable under free generation? For example, the in-domain ROUGE-L difference between SpectraLoom and T5-large is 0.38 points, while SpectraLoom’s reported seed standard deviation is 0.99. Please report paired seed-level results or confidence intervals and avoid interpreting small mean differences without uncertainty-aware comparison.
-
Generated-output analysis: What are the most frequent generated sentences, and what fraction of outputs belongs to the top 1, 5, and 10 generated strings? Given the very low uniqueness rates, qualitative examples and frequency statistics are necessary to determine whether metric gains reflect improved conditioning or a better corpus-level default prediction.
-
Biological interpretation of SBA: Are the learned band weights consistent across seeds, subjects, tasks, or words, and do they correspond to reproducible spectral patterns? The current ablation supports the utility of input-dependent gating, but does not establish that the gate discovers meaningful neurophysiological frequency selectivity rather than functioning as generic feature attention.