acceptodds
Under review as a conference paper at ICLR 2027

Beyond Aggregate Gains: Evaluating Robust Cross-Subject EEG-to-Text Retrieval

Abstract

Cross-subject EEG-to-text retrieval aims to recover linguistic content from in- dividuals unseen during model development, yet improvements in aggregate re- trieval performance do not necessarily indicate consistent benefits across unseen subjects. This distinction is especially important when architectural effects are comparable in magnitude to variability across subjects and stochastic training runs. We conduct a controlled evaluation under a leave-one-subject-out setting with training-only normalization, subject-disjoint model selection, canonical retrieval over 400 unique sentence contents, and three fixed training seeds. On a cleaned five-subject ZuCo sentence-reading setting, we compare a Graph-Free EEG Trans- former with prototype-based subject interaction, multi-head low-rank adaptation (MHLR), a Controlled BPR-style dense EEG–text baseline, and a deterministic Ridge control. The Graph-Free reference achieves 0.544 ± 0.407% Unique-Sentence Top-1 and 2.549 ± 0.855% Unique-Sentence Top-5 accuracy. Prototype-Only SIG attains the highest descriptive means (0.576 ± 0.293% and 2.765 ± 0.828%), but its paired gains over Graph-Free are only +0.032 and +0.216 percentage points and vary in both magnitude and direction across held-out subjects and training seeds. MHLR exhibits similarly heterogeneous effects, while the Controlled BPR-style and Ridge baselines do not outperform the Graph-Free reference under the same retrieval task. These experiments provide a concrete case in which a positive aggregate cross- subject retrieval gain does not translate into a consistent improvement across un- seen subjects and stochastic training trajectories. The results motivate evaluat- ing cross-subject EEG–language retrieval with subject-level effects and stochastic replication alongside aggregate performance.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG-to-text retrieval, cross-subject generalization, brain-computer interfaces, neural decoding, multimodal representation learning, robustness, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_2NLYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper evaluates text retrieval based on EEG signals. It references previous issues with relevant retrieval-based work. A few models are compared on a single, five-subject dataset. Its primary point is that better aggregated retrieval performance does not guarantee a model is better on every test subject. While its conclusion on the importance of reporting subject-level effects and variability across seeds is appreciated, it is also already widely recognized as a general phenomenon across science and statistics, and the phenomenon itself is only evaluated and established here with an absolute minimal level of effort.

Critical strengths

None

Critical weaknesses
  1. The paper does not position itself with respect to the literature and does not set out to solve a question or problem that goes beyond a simple ablation. Its contribution is very minimal.
Questions for authors

This work is so far below the bar that I don't foresee a possibility of a productive discussion and a potential recommendation for acceptance.

Reviewer_aizdOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether aggregate improvements in cross-subject EEG-to-text retrieval also lead to consistent improvements across unseen subjects and training seeds. The authors conduct a controlled evaluation on a five-subject ZuCo dataset, using leave-one-subject-out evaluation, training-only normalization, subject-disjoint model selection, a fixed bank of 400 unique sentences, and three training seeds. Several models are compared, including Graph-Free, Prototype-Only SIG, MHLR, a controlled BPR-style baseline, and Ridge. The main observation is that SIG achieves a small positive average gain over Graph-Free, but its effects are not consistent across subjects and seeds. Although more careful cross-subject evaluation has some value, and the authors provide descriptive evidence for their observation, I find the main scientific question and its takeaway rather limited. The weak empirical evidence and unresolved experimental issues further reduce my confidence in the significance and reliability of this work.

Critical strengths

The paper makes a reasonable effort to control evaluation conditions, including subject-disjoint training and validation, a fixed retrieval candidate bank, and matched subject–seed comparisons (Sec. 3.1–3.4, Appendix A). The authors also report subject-level effects instead of relying only on average accuracy (Fig. 2, Fig. 3, Table 2). I appreciate that they explicitly distinguish the 15 subject–seed results from 15 independent biological observations (Sec. 3.4 and Sec. 7.1). The provided supplementary code and experimental configurations are also useful for examining the reported results.

Critical weaknesses
  1. Limited novelty and scientific significance. The main conclusion, that a positive average gain does not necessarily mean consistent improvement across subjects or random seeds, seems rather obvious to me. I do not see a sufficiently new scientific insight beyond this expected observation. The paper mainly combines standard evaluation controls and reports paired differences, without establishing a substantially new evaluation principle or deeper understanding of cross-subject generalization. I learned little new from this paper. The distinction from existing evaluation and reliability studies is also not sufficiently established (Introduction, Sec. 2.3, Sec. 3.4, and Sec. 7.1).

  2. The empirical evidence is weak, and the actual retrieval signal needs stronger validation. In Table 1, Graph-Free achieves only 0.544% Top-1 accuracy compared with the 0.25% chance level. SIG improves this by just +0.032 percentage points, with a Top-5 improvement of +0.216 points. These are very small effects. More importantly, I do not see sufficient negative controls, such as shuffled EEG–text correspondence, noise inputs, or non-semantic controls, to establish what information is actually supporting retrieval. The experiments are also restricted to five biological subjects from one cleaned ZuCo setting. This makes the scientific value and generality of the observed heterogeneity rather limited (Sec. 5.1, Table 1, Fig. 2, Sec. 7.2).

  3. The preprocessing provenance raises a potentially important concern about the claimed strict evaluation protocol. The paper repeatedly emphasizes training-only normalization and strict separation of held-out subjects. However, the supplementary preprocessing code includes a subject-level Euclidean Alignment procedure that uses all available samples from a subject before the outer split. The provided repository also states that the exact historical preprocessing provenance has not been verified. I cannot confirm that this procedure was used to generate the reported experimental data, but this uncertainty matters because the strict protocol is a central contribution. The authors need to clarify the actual data preparation used in the experiments (Sec. 3.2, Appendix A.2, Reproducibility Statement, supplementary src/preprocess.py and README.md).

  4. Several experimental comparisons are not sufficiently isolated. First, Graph-Free/SIG uses a single-positive contrastive objective with a FIFO text bank, although sentence contents are repeated across instances. This can introduce false negatives and deserves further checking. Second, SIG constructs subject prototypes from the current minibatch, so its test predictions may depend on how queries are grouped. There is no clear batch-composition sensitivity analysis. In addition, MHLR uses unsupervised test-subject adaptation, while the BPR-style baseline uses a different text representation space from Graph-Free. The paper acknowledges some of these differences, but I am not convinced that the comparisons isolate architectural effects sufficiently (Sec. 4.2–4.4, Sec. 7.2, Appendix B.1/B.3/B.4, supplementary implementation).

  5. The presentation and related-work discussion need substantial improvement. I found the Abstract and Introduction repetitive and unnecessarily difficult to follow, especially given the relatively simple central question. Figure 1 is also difficult to read because of its small labels and crowded layout. The related-work coverage is quite limited, with only 13 references, and the discussion does not clearly establish the added scientific value over prior EEG decoding evaluation and reliability studies. Overall, I do not think the current presentation makes a convincing case for the significance of the work (Abstract, Introduction, Sec. 2, Fig. 1).

Questions for authors
  1. What is the main new scientific insight beyond the expected observation that average gains can hide subject-level and seed-level variability? How does this work go beyond existing evaluation and reliability studies?

  2. Can the authors provide stronger negative controls, such as shuffled EEG–text pairs or noise inputs, to show that the reported retrieval results genuinely depend on information in EEG rather than dataset-related cues?

  3. Can the authors clarify exactly how the processed EEG cache used in the reported experiments was generated? In particular, was subject-level Euclidean Alignment applied before the LOSO split, and were any statistics estimated using the held-out subject's data?

  4. Can the authors check whether repeated sentence targets introduce false negatives in the Graph-Free/SIG training objective, and whether SIG results change when test queries are grouped into different batches? Also, how should the differences in test-time adaptation and text representation spaces be considered when interpreting the model comparisons?

Reviewer_VaHcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at whether improvements in average cross-subject performance lead to improvements in individual unseen subjects in EEG-to-text retrieval. The paper performs a LOSO evaluation on a single dataset. Distinct validation subjects were used with a fixed bank of 400 unique sentence contents over 3 seeds. Multiple benchmarks were used, such as a graph free eeg transformer, ridge regression and MHLR.

The main finding is that prototype-Only SIG has the highest average retrieval performance, with effect size varying largely between seeds. The paper overall argues that Top-k gains should be reported alongside subject-level and stochastic replication when evaluating cross-subject EEG retrieval.

Critical strengths

Motivation and distinction between aggregate and robust improvements are presented clearly; the evaluation diagram and subject seed visualisations were also good.

Evaluation is done carefully and discusses common sources of optimistic cross-subject results. Good job at including multiple seeds.

Critical weaknesses

The overall evidence base is too small, there are only five biological subjects from one setting. The paper acknowledges but this reduces this work to a sort of case study rather than a comprehensive finding that would be deserving of an ICLR paper.

The effects of each architecture being interpreted are very small. The headline SIG improvements are only small gains in percentage points. It is very hard to claim this is a robust phenomenon about cross-subject architectures rather than just instability in a low-signal regime. Again, the participation count is very small, and this limits the significance of the central claim.

Absolute retrieval performance is also very low. This makes it difficult to know how meaningful the architectural differences are in practice.

While methodological correctness is acceptable, my main concern lies in the use of only one dataset with 5 subjects and extremely small architectural effects. The paper makes a useful methodological observation, but I do not believe the current paper supports a consequential ICLR contribution.

Questions for authors

How were the five included subjects and the ‘cleaned’ dataset derived? How sensitive are the findings to those preprocessing decisions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.