- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper audits sentence-retrieval evaluation in EEG-to-text systems. It argues that sentence-held-out splits overstate generalization because every reader remains represented in training. Under five leave-six-reader-out ZuCo folds and 24-way duration-matched retrieval pools, the submission reports top-1 accuracy falling from 0.1090 for reader-seen evaluation to 0.0458 for unseen readers, close to the 0.0417 chance level. It then introduces an identity-initialized per-reader residual channel adapter fitted with labelled target-reader trials and reports recovery toward the seen-reader ceiling, outperforming full fine-tuning at matched label counts.
Overall, the paper contributes: (i) a reader-held-out evaluation protocol for EEG-to-text retrieval; (ii) evidence that most performance under the conventional seen-reader protocol does not transfer to unseen readers; (iii) a compact per-reader calibration mechanism; and (iv) a careful analysis of calibration cost, architecture dependence, cross-corpus behaviour, and ocular or montage-related confounds.
Critical strengths
-
A consequential evaluation correction: the reader-disjoint protocol directly targets the generalization regime relevant to deployment and controls candidate-pool difficulty.
-
Strong diagnostic experimentation: the adapter controls, alternate architectures, high-pass analysis, and second-corpus replication are unusually thorough and transparent.
Critical weaknesses
-
Causal overinterpretation plus uncertain inference unit: the experiments show failure of cross-reader transfer, not an information decomposition into reader identity versus language; pool-level bootstrapping may understate uncertainty across 30 readers.
-
Adapter mechanism and practical value remain unresolved: the adapter may learn session/cap/artifact mappings rather than stable reader neurophysiology, and 400 labelled trials is a substantial calibration requirement.
-
“Reader identity” is stronger and more specific than the experiments establish. Holding out a reader shows that the model depends on reader-correlated information and that this dependence is not repaired by the tested label-free normalisations. It does not by itself separate neural idiosyncrasy from head geometry, electrode placement, impedance, recording-session effects, ocular behaviour, or other reader-correlated acquisition factors.
Questions for authors
-
Can you report per-reader effects and a reader-clustered or hierarchical bootstrap/permutation CI for the seen–unseen difference and the 94%/96% lost-margin statistic, including seed uncertainty?
-
Are adapter calibration, validation/model selection, and final retrieval pools strictly stimulus-disjoint? Additionally, can ChineseEEG support training an adapter on one novel/session and testing it on the other for the same reader? This would distinguish stable reader calibration from same-session acquisition mapping.
-
What exactly is included in the “94%” and “96%” attribution to reader identity? Do the authors intend “identity” to mean any stable reader-correlated signal, including recording and behavioural nuisance factors, or specifically a physiological reader signature?
2 comments
Response to Reviewer Shur(Q3)
Q3. What does "reader identity" include?
We mean it operationally: any signal that a model trained on other readers cannot use for a new reader. That includes recording and behavioural nuisance factors (head geometry, electrode placement, impedance, session state, ocular behaviour) as well as any physiological signature. Our experiments do not separate these, and the cross-session result shows that at least the recoverable part is session-bound. The revision:
- replaces "reader identity" with "reader- and session-correlated factors" in the title, abstract and claims, and defines the term operationally where it is first used;
- states that the 95% is the share of the seen-reader margin above chance that is lost when the reader is unseen, and not an information decomposition into reader versus language;
- lists the candidate sources, notes which ones our controls address (data volume, label-free normalisation, 4 Hz high-pass, cross-session transfer) and which remain open (cap placement, impedance, behavioural differences).
Thank you again. Q2 in particular produced the most informative new result in the revision.
Response to Reviewer Shur(Q1&Q2)
Thank you for the careful review. All three of your questions led to new analyses, and the answer to Q2 changes how we interpret the adapter. The shared tables are in the general response.
Q1. Per-reader effects and reader-clustered / hierarchical CIs, including seed uncertainty.
We reassigned every retrieval decision to the reader whose trial was the query and redid the inference with readers as the unit (20,000 bootstrap resamples), plus a two-level reader × seed bootstrap.
The submission's 94% (ZuCo) and 96% (ChineseEEG) were pool-weighted. Weighting readers equally gives 95% on both. With non-overlapping pools (stride 24) the figures are 94% [78%, 109%] and 95% [87%, 102%]. The revision reports the reader × seed intervals as the main ones and adds per-reader plots for every arm in the appendix. You were right that the pool-level CIs understated uncertainty. The reader-level intervals are wider, and the conclusion holds under them.
Q2a. Are calibration, model selection and test pools stimulus-disjoint?
Yes for the data, and we checked it directly. Sentence IDs are disjoint across train / val / test on both corpora (ZuCo 1,167 / 180 / 182, ChineseEEG 7,982 / 998 / 998; zero overlap in every pair). A held-out reader's calibration trials are drawn only from that reader's training-split sentences, and scoring uses only test-split pools. Calibration runs use a fixed 20 epochs, so validation trials play no role in stopping or checkpoint choice.
One point needs disclosure. The adapter learning rate was chosen from {1e-3, 3e-3} on ZuCo by comparing test-pool scores on folds 0–2. At N=400 the adapter scored 0.1031 at 1e-3 and 0.1094 at 3e-3 on those folds. Full fine-tuning scores 0.0907 over all five folds, so the adapter stays ahead at either learning rate. The revision states this selection step in full. All ChineseEEG runs reuse the ZuCo value (3e-3) without further tuning, so the second corpus works as an untuned replication.
Q2b. Train the adapter on one novel and test on the other.
We ran this for the 9 ChineseEEG readers who read both novels (30 runs, N=400, three seeds):
The adapter does not transfer across sessions for the same reader. Your question separates the two readings, and the data support the second: the adapter learns a mapping specific to the session (or the session's stimulus), not a stable reader calibration. This also bears on your Weakness 2. Because the calibration has to be repeated per session, its practical value as a calibrate-once BCI step is weaker than the submission implied. The revision says so directly (Sec. 6.2, Sec. 7, Sec. 9) and presents the adapter as a diagnostic: it shows that the missing signal is recoverable by a linear per-session map in front of a frozen encoder.
Two caveats from the submission still hold, and the revision states them alongside this result (Sec. 6, "ocular" paragraph; Appendix E, Table 11). The reader effect survives a 4 Hz high-pass on ZuCo (92% of the margin lost), so it is not mainly the low-frequency frontal component that we attribute to ocular activity. After the filter, the adapter at N=400 still matches the filtered seen-reader score (0.0936 vs 0.0960, n.s.). That result comes from joint calibration as in the submission, and the revision labels it so.
On the 400-label cost: on ZuCo the calibration curve is at 0.0588 (1.41× chance) with N=25 and recovers about half the margin by N=100 (0.0780). Those budgets were run with joint calibration. The independent runs cover N=400 only, where the adapter recovers 87% on ZuCo and 40% on ChineseEEG. The revision reports the curve with that caveat (Fig. 1(b), Table 3 and Table 5 captions, Sec. 8).