- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper mainly investigates what signal properties actually drive the prediction of EEG foundation models (EEG-FMs) behind the downstream task score, and how to attribute the observed model behaviors to either the downstream interface (adapter/readout) or the pretrained backbone. To solve the confounding problem in existing benchmarks, the authors propose a two-stage diagnostic framework on 5 released EEG-FMs across 4 tasks. Stage I fixes the adapter, frozen backbone, and a unified linear readout. Stage II applies 8 input perturbation protocols across spatial, temporal-spectral, and artifact/background dimensions with paired controls, calibrated on synthetic EEG. The results show that current EEG-FMs rely heavily on coarse signal properties (montage order, short-window energy, low-frequency delta band, and channel RMS scale) rather than fine physiological structures. Finally, by refitting only the readout (e.g., with mirror augmentation), the authors separate what the readout left unused from true backbone defects.
Critical strengths
- Rigorous paired-control design: The paper isolates target physiological interventions from operator side effects very well (e.g., equal-channel scattered masking vs. regional masking, repeat-fill vs. zero-fill truncation, and energy-only attenuation vs. noise crossfade), avoiding the false attribution in previous perturbation studies.
- Planted-ground-truth calibration: Since real EEG lacks ground truth for feature importance, validating the protocols on synthetic EEG with planted dependencies and 3 fault injections (shuffled channels, wrong sampling rate, independent labels) makes the diagnostic pipeline very convincing.
- Sobering empirical findings on shortcut learning: It clearly reveals that high task scores conceal severe shortcuts—for example, masking central sensorimotor electrodes (C3/Cz/C4) barely hurts motor imagery, while CBraMod and CSBrain still get ~0.90 AUROC on seizure detection even when the waveform is 100% replaced by RMS-matched pink noise.
- Closed-loop attribution via readout refits: By freezing the backbone and only refitting the linear readout with mirror or artifact augmentation against an equal-budget control, the paper provides a practical way to tell whether a missing property is absent in the backbone or just ignored by the linear head.
Critical weaknesses
- Small test subject size and single dataset per task: Each task only uses one cohort, and the test sets for Seizure (Siena) and Emotion (SEED-IV) only have N=3 test subjects. This makes the statistical power across subjects relatively weak and potentially sensitive to dataset-specific bias.
- Restricted to frozen backbone + linear readout regime: The paper only evaluates one state family (frozen backbone with affine readout). If fine-grained physiological rhythms are non-linearly entangled in the representation, a simple linear probe cannot read them out, which may underestimate the backbone capability compared to MLP/attention heads or full fine-tuning.
- Floor effects on certain task-model pairs: On Emotion recognition (SEED-IV), the clean lift L0 of all 5 models is near zero (-0.003 to 0.068, close to 0.50 chance level), and BIOT also has L0=0.011 on Motor. Because of this floor effect, the Emotion task provides very limited valid diagnostic information across the protocols.
- Heavy notation and metric system: The paper defines more than 20 custom abbreviations and derived coordinates (RIS, SCS, PRR, AD, APG, MRA, CAD, EET, etc.), which increases the reading burden for readers to track the main message.
Questions for authors
- About non-linear readout or fine-tuning: Since linear readout can only extract linearly separable features, could the weak response to fine physiological rhythms (like mu/beta in Motor or alpha microstructure in S2.5) be caused by the linear head limitation? Can the authors supplement an experiment using a lightweight non-linear head (e.g., MLP or attention pooling) or fine-tuning on 1-2 representative protocols (such as S1.1, S1.2, or S2.3)?
- About the N=3 test subjects in Seizure and Emotion: Given only 3 test subjects in Siena and SEED-IV, how stable are the diagnostic profiles across different train/val/test subject splits, or on another dataset with more subjects (e.g., TUH Seizure)?
- About the floor effect in Emotion recognition: Since all 5 models perform near chance on SEED-IV under the current multi-task joint linear training (Stage I), does training a task-specific readout separately (or using another emotion/clinical task with higher clean lift L0) make the Emotion diagnostic profile observable?
Overall, the key value of this paper lies in its controlled causal diagnostic design, and the limitations above do not affect its strong positive value. If the authors can supplement experiments on (1) non-linear readout/fine-tuning and (2) more test subjects/splits during the rebuttal stage, I will be happy to raise my score.
3 comments
Response to Reviewer FrXa (2/3): Q2
Q2. Are the Seizure findings stable beyond the three original test subjects?
Five-fold subject-disjoint evaluation across all 14 Siena subjects supports a narrower conclusion: CBraMod and CSBrain retain substantial discrimination after full waveform replacement, but the original post-replacement AUROC increase does not persist. Zero-fill sensitivity also remains substantial (Tables I.6–I.8; original three-subject limit: L466–468).
We refitted a frozen-backbone linear readout in each of five folds (single seed, 20260528), so every Siena subject is held out once (Appendix I.2, Tables I.6–I.8). The diagnostic interventions were applied under the corresponding evaluation routes.
Table I.7 — Seizure AUROC (clean → full channel-RMS-matched waveform replacement). Original three-subject figures are three-seed means; the right column is subject-disjoint evaluation over all 14 subjects.
| Model | Original 3-subject test | 14-subject CV |
|---|---|---|
| CBraMod | 0.887 → 0.905 | 0.812 → 0.709 |
| CSBrain | 0.870 → 0.894 | 0.919 → 0.827 |
| BIOT | 0.811 → 0.485 | 0.768 → 0.477 |
| LaBraM | 0.652 → 0.518 | 0.640 → 0.549 |
On all 14 subjects, full replacement reduces CBraMod and CSBrain AUROC, rather than increasing it as on the original split (L391–392). Yet they retain 67% and 78% of clean lift; paired AUROC losses are 0.103 [0.047, 0.172] and 0.092 [0.059, 0.123] (95% intervals; Table I.8). Thus, the score increase was split-sensitive, while substantial above-chance discrimination survives without the original waveform. BIOT and LaBraM show larger losses, without proving exclusive waveform dependence.
The aperture result is also similar across Siena partitions: CBraMod zero-fill loss is 0.453 (§4.3, L361–362) versus 0.456, with repeat fill recovering 97.3% versus 92.7% (Table I.6). This establishes fill-policy sensitivity, not a uniquely isolated boundary artifact.
Of your two options, we chose subject-disjoint re-splitting to vary training and test subject membership while holding the Siena acquisition, montage, and annotation setting fixed. TUH Seizure would additionally introduce differences in acquisition, montage, and annotation, making it harder to isolate the effect of subject composition. This analysis therefore tests sensitivity to the Siena subject partition, not external clinical generalization. We do not claim TUH validation. Severe 0.1× attenuation (L393–395) is also outside the training-amplitude range for most Siena windows (Table I.9); the revised amplitude claim rests on RMS-matched waveform replacement, not on below-chance attenuation scores.
Response to Reviewer FrXa (3/3): Q3 and W4
Q3. Can an Emotion-specific readout overcome the near-chance floor?
Not in the evaluated frozen-backbone routes. Emotion-only linear training changes held-out AUROC by at most 0.008, while full fine-tuning increases test AUROC to about 0.60 in three models. The canonical near-chance cells remain unsuitable for positive physiological-robustness claims.
We trained an Emotion-only linear head with its own validation checkpoint and compared it with joint linear training, nonlinear alternatives and full fine-tuning (Appendix I.3, Tables I.10–I.11).
Table I.10 — Emotion held-out AUROC (chance 0.50). The first two states keep the backbone frozen; the fine-tuning results are added-state runs.
| Model | Joint linear | Emotion-only linear | Full fine-tuning |
|---|---|---|---|
| BIOT | 0.487 | 0.494 | 0.539 |
| CBraMod | 0.531 | 0.527 | 0.600 |
| CSBrain | 0.556 | 0.554 | 0.556 |
| LaBraM | 0.562 | 0.563 | 0.601 |
| REVE | 0.557 | 0.554 | 0.600 |
Emotion-only linear training changes held-out AUROC by at most 0.008 across the five models (Table I.10). Full fine-tuning raises CBraMod, LaBraM and REVE to 0.600, 0.601 and 0.600, respectively (gains of 0.069, 0.039 and 0.043 over joint frozen-linear scores). The improvement is not universal: CSBrain remains at 0.556, and BIOT reaches only 0.539.
The alternative frozen nonlinear heads likewise do not yield consistently strong held-out discrimination (Table I.10). Four of five task-specific frozen heads fit training subjects well (final training balanced accuracy about 0.93–1.00), yet validation/test scores remain much lower (Table I.11), consistent with limited cross-subject transfer rather than a demonstrated absence of information in the backbone. The improved fine-tuned scores show that the evaluated frozen route is not an upper bound, but do not isolate which adaptation component caused the gain or establish a reliable Emotion perturbation profile. Emotion still has only three original test subjects and no independent cohort; flat canonical perturbation curves are therefore not interpreted as physiological robustness, as already set out by Step 1 of §3.3 (L200–202) and Table C.7 (L1678).
- W4, notation: We reduced specialized abbreviations in the main text (introduced in §3.3, L191–197), using descriptive terms such as “clean lift,” “regional loss,” and “loss beyond scattered masking”; Appendix B retains the formal definitions, and Table I.36 provides the terminology map.
Response to Reviewer FrXa (1/3): Q1
Thank you for the constructive review and for identifying the two experiments most relevant to reassessing our conclusions. We prioritized both: nonlinear readouts/full fine-tuning on Motor (Tables I.1–I.3) and five-fold subject-disjoint evaluation across all 14 Siena subjects (Tables I.6–I.8). The added tests show that the measured Motor μ/β response remains weak in the evaluated states, whereas the original Seizure AUROC increase after waveform replacement is split-sensitive. We also tested an Emotion-specific readout and a second Motor cohort (BCI Competition IV-2a; Table I.16). We retained the core perturbation operators while correcting the S2.3 spectral reference of the fixed-scale models to self-RMS matching. We answer Q1–Q3 in order, with related weaknesses under each. Tables I.x refer to Appendix I of the Revised Manuscript; changes are marked in blue.
Line numbers (L) refer to the Original Submission; "now" refers to the Revised Manuscript.
Q1. Do nonlinear heads or full fine-tuning change the physiological readings?
The weak measured central-region and μ/β responses persist in the tested additional states. Full fine-tuning improves Motor discrimination and increases the left/right decision-reversal response, but these observations do not establish that the pretrained backbone lacks finer physiological information.
We agree that a frozen linear readout does not characterize all information potentially encoded in the backbone. We therefore reran the Motor protocols in two states already declared in Table A.5 (L874–885): (i) a nonlinear head on the frozen backbone and (ii) full backbone/readout fine-tuning for 30 epochs, with the same data and splits (Tables I.1–I.3).
Across the four Motor models with meaningful clean lift (CBraMod, CSBrain, LaBraM and REVE; BIOT's clean lift of 0.011 is set aside by Step 1 of §3.3, L200–202), full fine-tuning raises clean AUROC from 0.689–0.764 to 0.809–0.847, yet the μ/β band-stop loss remains within −0.03 to +0.03 AUROC, versus −0.02 to +0.01 in the canonical state (below 2% in §4.3, L368–369). Central-region masking relative to equal-channel scattered controls also remains nonpositive (as in §4.3, L321–323), whereas left/right decision reversal under the hemispheric swap rises from 0.27–0.47 to 0.54–0.67 (Tables I.1–I.3). Stronger lateralization accompanies, but is not proven to cause, the score gain.
Table I.2 — Motor responses across evaluated states. Ranges over four readable routes; added states are principally single-seed. Reversal is a fraction; other entries are AUROC losses or regional-minus-control differences.
The frozen nonlinear-head test gives a complementary answer. With flattened-feature MLPs, CBraMod/CSBrain/LaBraM achieve Motor AUROC 0.745/0.744/0.726 (canonical linear: 0.751/0.753/0.689); their μ/β losses remain small. REVE uses attention pooling and achieves 0.671 rather than its canonical 0.764, so we do not treat every nonlinear configuration as equally observable. S2.5 was evaluated under full fine-tuning, not under the nonlinear head (Tables I.1–I.3).
Head architecture matters: global attention pooling over channels and time takes CBraMod/CSBrain/LaBraM close to chance (0.513–0.520), while flattened MLP heads regain AUROC 0.726–0.745 (Tables I.1, I.4). Thus, the spatial organization retained at the interface matters, not merely whether the head is nonlinear.
The added-state tests weaken a linear-head-only explanation, but do not prove that finer physiological information is absent from the pretrained backbone. Untested heads, objectives and states remain possible. Conversely, mirror-augmented readout refitting against an equal-budget control improves decision-level reflection consistency with the backbone frozen (§5.1, L411–417). We therefore report route-level diagnoses, not a universal upper bound on backbone capability; §5.2 treats backbone changes as hypotheses.