acceptodds
Under review as a conference paper at ICLR 2027

Dissecting the Score: Calibrated Diagnosis of EEG Foundation Models

Abstract

EEG foundation models (EEG-FMs) continue to achieve strong performance on downstream tasks, but task scores alone do not reveal which properties of the signal drive their predictions. In particular, it remains unclear whether predictions depend on physiologically meaningful features of EEG or on simpler properties such as electrode arrangement and signal amplitude. Input perturbations do not fully resolve this question, because their effects can also depend on the adapter, the readout, and artifacts introduced by the perturbation itself. We therefore propose a cross-architecture diagnostic benchmark that uses neurophysiology as a common reference. For each model, we first fix the adapter, pretrained backbone, and readout, and then apply eight diagnostic protocols to the input without further training. Paired controls are used to separate the intended intervention from major side effects, and three representative protocols are validated on synthetic EEG with known signal dependencies. Across five released models and four tasks, we observe a recurring pattern: predictions are often more sensitive to global or low-level signal properties than to the specific brain regions and EEG rhythms expected to be relevant to the task. On motor imagery, swapping corresponding electrodes between the two hemispheres reduces AUROC by 0.18-0.28, whereas completely masking the predefined central sensorimotor region changes performance by less than 0.04. On seizure detection, two models that preserve input amplitude remain highly predictive even after the original waveform is fully replaced by RMS-matched background, yet fall below chance when the original waveform is kept but strongly attenuated. The same design also allows us to test where these behaviors can be changed. With the pretrained backbone kept frozen, a mirror-augmented readout refit makes the motor imagery predictions respond more consistently to left-right electrode swapping. This shows that some behaviors revealed by the diagnostic tests can be changed through the downstream interface without modifying the pretrained backbone. Overall, the benchmark provides a controlled way to uncover signal dependencies that are hidden by task scores and to test whether they can be modified at the downstream stage.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG foundation models, benchmark, diagnostic evaluation, calibration, controlled perturbation, neurophysiology

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FrXaOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper mainly investigates what signal properties actually drive the prediction of EEG foundation models (EEG-FMs) behind the downstream task score, and how to attribute the observed model behaviors to either the downstream interface (adapter/readout) or the pretrained backbone. To solve the confounding problem in existing benchmarks, the authors propose a two-stage diagnostic framework on 5 released EEG-FMs across 4 tasks. Stage I fixes the adapter, frozen backbone, and a unified linear readout. Stage II applies 8 input perturbation protocols across spatial, temporal-spectral, and artifact/background dimensions with paired controls, calibrated on synthetic EEG. The results show that current EEG-FMs rely heavily on coarse signal properties (montage order, short-window energy, low-frequency delta band, and channel RMS scale) rather than fine physiological structures. Finally, by refitting only the readout (e.g., with mirror augmentation), the authors separate what the readout left unused from true backbone defects.

Critical strengths
  1. Rigorous paired-control design: The paper isolates target physiological interventions from operator side effects very well (e.g., equal-channel scattered masking vs. regional masking, repeat-fill vs. zero-fill truncation, and energy-only attenuation vs. noise crossfade), avoiding the false attribution in previous perturbation studies.
  2. Planted-ground-truth calibration: Since real EEG lacks ground truth for feature importance, validating the protocols on synthetic EEG with planted dependencies and 3 fault injections (shuffled channels, wrong sampling rate, independent labels) makes the diagnostic pipeline very convincing.
  3. Sobering empirical findings on shortcut learning: It clearly reveals that high task scores conceal severe shortcuts—for example, masking central sensorimotor electrodes (C3/Cz/C4) barely hurts motor imagery, while CBraMod and CSBrain still get ~0.90 AUROC on seizure detection even when the waveform is 100% replaced by RMS-matched pink noise.
  4. Closed-loop attribution via readout refits: By freezing the backbone and only refitting the linear readout with mirror or artifact augmentation against an equal-budget control, the paper provides a practical way to tell whether a missing property is absent in the backbone or just ignored by the linear head.
Critical weaknesses
  1. Small test subject size and single dataset per task: Each task only uses one cohort, and the test sets for Seizure (Siena) and Emotion (SEED-IV) only have N=3 test subjects. This makes the statistical power across subjects relatively weak and potentially sensitive to dataset-specific bias.
  2. Restricted to frozen backbone + linear readout regime: The paper only evaluates one state family (frozen backbone with affine readout). If fine-grained physiological rhythms are non-linearly entangled in the representation, a simple linear probe cannot read them out, which may underestimate the backbone capability compared to MLP/attention heads or full fine-tuning.
  3. Floor effects on certain task-model pairs: On Emotion recognition (SEED-IV), the clean lift L0 of all 5 models is near zero (-0.003 to 0.068, close to 0.50 chance level), and BIOT also has L0=0.011 on Motor. Because of this floor effect, the Emotion task provides very limited valid diagnostic information across the protocols.
  4. Heavy notation and metric system: The paper defines more than 20 custom abbreviations and derived coordinates (RIS, SCS, PRR, AD, APG, MRA, CAD, EET, etc.), which increases the reading burden for readers to track the main message.
Questions for authors
  1. About non-linear readout or fine-tuning: Since linear readout can only extract linearly separable features, could the weak response to fine physiological rhythms (like mu/beta in Motor or alpha microstructure in S2.5) be caused by the linear head limitation? Can the authors supplement an experiment using a lightweight non-linear head (e.g., MLP or attention pooling) or fine-tuning on 1-2 representative protocols (such as S1.1, S1.2, or S2.3)?
  2. About the N=3 test subjects in Seizure and Emotion: Given only 3 test subjects in Siena and SEED-IV, how stable are the diagnostic profiles across different train/val/test subject splits, or on another dataset with more subjects (e.g., TUH Seizure)?
  3. About the floor effect in Emotion recognition: Since all 5 models perform near chance on SEED-IV under the current multi-task joint linear training (Stage I), does training a task-specific readout separately (or using another emotion/clinical task with higher clean lift L0) make the Emotion diagnostic profile observable?

Overall, the key value of this paper lies in its controlled causal diagnostic design, and the limitations above do not affect its strong positive value. If the authors can supplement experiments on (1) non-linear readout/fine-tuning and (2) more test subjects/splits during the rebuttal stage, I will be happy to raise my score.

3 comments
AuthorsOpenReview ↗

Response to Reviewer FrXa (1/3): Q1

Thank you for the constructive review and for identifying the two experiments most relevant to reassessing our conclusions. We prioritized both: nonlinear readouts/full fine-tuning on Motor (Tables I.1–I.3) and five-fold subject-disjoint evaluation across all 14 Siena subjects (Tables I.6–I.8). The added tests show that the measured Motor μ/β response remains weak in the evaluated states, whereas the original Seizure AUROC increase after waveform replacement is split-sensitive. We also tested an Emotion-specific readout and a second Motor cohort (BCI Competition IV-2a; Table I.16). We retained the core perturbation operators while correcting the S2.3 spectral reference of the fixed-scale models to self-RMS matching. We answer Q1–Q3 in order, with related weaknesses under each. Tables I.x refer to Appendix I of the Revised Manuscript; changes are marked in blue.

Line numbers (L) refer to the Original Submission; "now" refers to the Revised Manuscript.

Q1. Do nonlinear heads or full fine-tuning change the physiological readings?

The weak measured central-region and μ/β responses persist in the tested additional states. Full fine-tuning improves Motor discrimination and increases the left/right decision-reversal response, but these observations do not establish that the pretrained backbone lacks finer physiological information.

We agree that a frozen linear readout does not characterize all information potentially encoded in the backbone. We therefore reran the Motor protocols in two states already declared in Table A.5 (L874–885): (i) a nonlinear head on the frozen backbone and (ii) full backbone/readout fine-tuning for 30 epochs, with the same data and splits (Tables I.1–I.3).

Across the four Motor models with meaningful clean lift (CBraMod, CSBrain, LaBraM and REVE; BIOT's clean lift of 0.011 is set aside by Step 1 of §3.3, L200–202), full fine-tuning raises clean AUROC from 0.689–0.764 to 0.809–0.847, yet the μ/β band-stop loss remains within −0.03 to +0.03 AUROC, versus −0.02 to +0.01 in the canonical state (below 2% in §4.3, L368–369). Central-region masking relative to equal-channel scattered controls also remains nonpositive (as in §4.3, L321–323), whereas left/right decision reversal under the hemispheric swap rises from 0.27–0.47 to 0.54–0.67 (Tables I.1–I.3). Stronger lateralization accompanies, but is not proven to cause, the score gain.

Table I.2 — Motor responses across evaluated states. Ranges over four readable routes; added states are principally single-seed. Reversal is a fraction; other entries are AUROC losses or regional-minus-control differences.

ProbeFrozen linearFrozen nonlinearFull fine-tuning
Left/right decision reversal (S1.2)0.27–0.470.22–0.400.54–0.67
Central minus scattered masking (S1.1)−0.02 to −0.01−0.03 to −0.02−0.04 to −0.01
μ/β band-stop loss (S2.3)−0.02 to 0.01−0.02 to 0.01−0.03 to 0.03
α microstructure loss (S2.5)0.006–0.012Not tested0.007–0.011
Frontal minus scattered masking (S1.1)0.03–0.070.06–0.100.06–0.08
δ band-stop loss (S2.3)0.04–0.110.04–0.110.10–0.17

The frozen nonlinear-head test gives a complementary answer. With flattened-feature MLPs, CBraMod/CSBrain/LaBraM achieve Motor AUROC 0.745/0.744/0.726 (canonical linear: 0.751/0.753/0.689); their μ/β losses remain small. REVE uses attention pooling and achieves 0.671 rather than its canonical 0.764, so we do not treat every nonlinear configuration as equally observable. S2.5 was evaluated under full fine-tuning, not under the nonlinear head (Tables I.1–I.3).

Head architecture matters: global attention pooling over channels and time takes CBraMod/CSBrain/LaBraM close to chance (0.513–0.520), while flattened MLP heads regain AUROC 0.726–0.745 (Tables I.1, I.4). Thus, the spatial organization retained at the interface matters, not merely whether the head is nonlinear.

The added-state tests weaken a linear-head-only explanation, but do not prove that finer physiological information is absent from the pretrained backbone. Untested heads, objectives and states remain possible. Conversely, mirror-augmented readout refitting against an equal-budget control improves decision-level reflection consistency with the backbone frozen (§5.1, L411–417). We therefore report route-level diagnoses, not a universal upper bound on backbone capability; §5.2 treats backbone changes as hypotheses.

AuthorsOpenReview ↗

Response to Reviewer FrXa (2/3): Q2

Q2. Are the Seizure findings stable beyond the three original test subjects?

Five-fold subject-disjoint evaluation across all 14 Siena subjects supports a narrower conclusion: CBraMod and CSBrain retain substantial discrimination after full waveform replacement, but the original post-replacement AUROC increase does not persist. Zero-fill sensitivity also remains substantial (Tables I.6–I.8; original three-subject limit: L466–468).

We refitted a frozen-backbone linear readout in each of five folds (single seed, 20260528), so every Siena subject is held out once (Appendix I.2, Tables I.6–I.8). The diagnostic interventions were applied under the corresponding evaluation routes.

Table I.7 — Seizure AUROC (clean → full channel-RMS-matched waveform replacement). Original three-subject figures are three-seed means; the right column is subject-disjoint evaluation over all 14 subjects.

ModelOriginal 3-subject test14-subject CV
CBraMod0.887 → 0.9050.812 → 0.709
CSBrain0.870 → 0.8940.919 → 0.827
BIOT0.811 → 0.4850.768 → 0.477
LaBraM0.652 → 0.5180.640 → 0.549

On all 14 subjects, full replacement reduces CBraMod and CSBrain AUROC, rather than increasing it as on the original split (L391–392). Yet they retain 67% and 78% of clean lift; paired AUROC losses are 0.103 [0.047, 0.172] and 0.092 [0.059, 0.123] (95% intervals; Table I.8). Thus, the score increase was split-sensitive, while substantial above-chance discrimination survives without the original waveform. BIOT and LaBraM show larger losses, without proving exclusive waveform dependence.

The aperture result is also similar across Siena partitions: CBraMod zero-fill loss is 0.453 (§4.3, L361–362) versus 0.456, with repeat fill recovering 97.3% versus 92.7% (Table I.6). This establishes fill-policy sensitivity, not a uniquely isolated boundary artifact.

Of your two options, we chose subject-disjoint re-splitting to vary training and test subject membership while holding the Siena acquisition, montage, and annotation setting fixed. TUH Seizure would additionally introduce differences in acquisition, montage, and annotation, making it harder to isolate the effect of subject composition. This analysis therefore tests sensitivity to the Siena subject partition, not external clinical generalization. We do not claim TUH validation. Severe 0.1× attenuation (L393–395) is also outside the training-amplitude range for most Siena windows (Table I.9); the revised amplitude claim rests on RMS-matched waveform replacement, not on below-chance attenuation scores.

AuthorsOpenReview ↗

Response to Reviewer FrXa (3/3): Q3 and W4

Q3. Can an Emotion-specific readout overcome the near-chance floor?

Not in the evaluated frozen-backbone routes. Emotion-only linear training changes held-out AUROC by at most 0.008, while full fine-tuning increases test AUROC to about 0.60 in three models. The canonical near-chance cells remain unsuitable for positive physiological-robustness claims.

We trained an Emotion-only linear head with its own validation checkpoint and compared it with joint linear training, nonlinear alternatives and full fine-tuning (Appendix I.3, Tables I.10–I.11).

Table I.10 — Emotion held-out AUROC (chance 0.50). The first two states keep the backbone frozen; the fine-tuning results are added-state runs.

ModelJoint linearEmotion-only linearFull fine-tuning
BIOT0.4870.4940.539
CBraMod0.5310.5270.600
CSBrain0.5560.5540.556
LaBraM0.5620.5630.601
REVE0.5570.5540.600

Emotion-only linear training changes held-out AUROC by at most 0.008 across the five models (Table I.10). Full fine-tuning raises CBraMod, LaBraM and REVE to 0.600, 0.601 and 0.600, respectively (gains of 0.069, 0.039 and 0.043 over joint frozen-linear scores). The improvement is not universal: CSBrain remains at 0.556, and BIOT reaches only 0.539.

The alternative frozen nonlinear heads likewise do not yield consistently strong held-out discrimination (Table I.10). Four of five task-specific frozen heads fit training subjects well (final training balanced accuracy about 0.93–1.00), yet validation/test scores remain much lower (Table I.11), consistent with limited cross-subject transfer rather than a demonstrated absence of information in the backbone. The improved fine-tuned scores show that the evaluated frozen route is not an upper bound, but do not isolate which adaptation component caused the gain or establish a reliable Emotion perturbation profile. Emotion still has only three original test subjects and no independent cohort; flat canonical perturbation curves are therefore not interpreted as physiological robustness, as already set out by Step 1 of §3.3 (L200–202) and Table C.7 (L1678).

  • W4, notation: We reduced specialized abbreviations in the main text (introduced in §3.3, L191–197), using descriptive terms such as “clean lift,” “regional loss,” and “loss beyond scattered masking”; Appendix B retains the formal definitions, and Table I.36 provides the terminology map.
Reviewer_ahivOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a two-stage diagnostic benchmark to evaluate what signal properties drive downstream predictions in five pretrained EEG foundation models (BIOT, CBraMod, CSBrain, LaBraM, REVE) across four tasks (Motor Imagery, Sleep Staging, Seizure Detection, Emotion Recognition). In Stage I, each model is locked into a canonical evaluation route pairing its native preprocessing adapter, frozen pretrained backbone, and a single linear readout layer trained across three seeds. In Stage II, eight input-side perturbation protocols—spanning spatial organization (S1.1 regional vs. scattered masking, S1.2 homologous left$\leftrightarrow$right swap), temporal–spectral structure (S2.1 zero-fill vs. repeat-fill aperture truncation, S2.3 low-pass/band-stop filtering, S2.5 alpha rhythm modulation), and artifact/background dynamics (S3.2 EMG/EOG injection, S3.3 noise crossfade vs. amplitude attenuation, S3.4 non-brain rejection entropy)—probe the locked routes alongside synthetic planted-signal calibration (Appendix E) and closed-loop readout refits on frozen backbones (Appendix F). The study finds that high leaderboard scores often rely on coarse shortcuts—such as raw channel amplitude scale on Seizure (CBraMod and CSBrain reach ∼0.90\sim 0.90 AUROC on pure RMS-matched pink noise while falling below chance under 0.1×0.1\times amplitude scaling), slow δ\delta-band power, and rigid left–right electrode slot indexing—and demonstrates via mirror-augmented and artifact-augmented readout refits that several failures (S1.2 on all models, S3.2 on LaBraM/BIOT) stem from the linear readout interface rather than the frozen backbone.

Critical strengths
  1. Rigorous Diagnostic Design with Paired Negative/Confound Controls and Closed-Loop Readout Refits: Pairing every primary perturbation with matched side-effect controls (e.g., regional vs. equal-channel scattered masking in S1.1, zero-fill vs. repeat-fill in S2.1, and noise crossfade vs. pure amplitude attenuation in S3.3) and following up with closed-loop readout refits on bit-identical frozen backbones (Appendix F) provides a principled methodology for separating perturbation side effects and linear-readout interface flaws from true backbone deficits.
  2. Actionable Empirical Insights (Especially the S3.3 Amplitude Shortcut and F.1/F.2 Interface Attribution): Demonstrating that fixed-scale evaluation routes (CBraMod, CSBrain) achieve 0.9050.905 and 0.8940.894 Seizure AUROC when 100% of the EEG waveform is replaced by RMS-matched pink noise (α=1\alpha = 1)—yet collapse below chance (0.2640.264 and 0.3230.323) when true seizure waveforms are scaled by 0.1×0.1\times—exposes a critical evaluation vulnerability in current EEG-FM benchmarks. Likewise, showing that mirror-augmented readout refitting recovers contralateral motor symmetry on frozen backbones cleanly proves that the representation was present in the backbone all along.
Critical weaknesses
  1. Destructive Unit-RMS Reference Normalization in S2.3 (Table D.1) and Confounding of S2.1 Zero-Fill with S3.3 Window Energy / Unpooled Patch Slots:
    • Unit-RMS Reference Collapse in S2.3: In S2.3 (Frequency Sensitivity), the protocol disables adapter z-scoring and normalizes every channel to unit RMS (=1 μV= 1\,\mu\text{V}) before the adapter's fixed ×0.01\times 0.01 scaling. Because real EEG has RMS ∼10–50 μV\sim 10\text{--}50\,\mu\text{V}, this scales down inputs to fixed-scale models by 10×–50×10\times\text{--}50\times. As shown in Table D.1 (p. 35), before any frequency filter is applied, this reference normalization alone destroys 83% of CBraMod's Seizure lift (0.887→0.5640.887 \to 0.564), 94% of CSBrain's Seizure lift (0.870→0.5220.870 \to 0.522), and 76–79% of their Sleep lift (0.411→0.2450.411 \to 0.245 and 0.544→0.2810.544 \to 0.281)! Normalizing each filtered channel x~c\tilde{x}_c to match its own pre-filter clean RMS ∥xcclean∥RMS\|x_c^{\text{clean}}\|_{\text{RMS}} (rather than 1.01.0) is needed to isolate spectral filtering from global amplitude attenuation.
    • Incomplete Attribution of S2.1 Zero-Fill vs. Repeat-Fill: Section 4.3 (lines 359–364) attributes the massive loss under zero-fill in S2.1 (e.g., CBraMod Seizure 2 s2\text{ s} zero-fill retention 0.0810.081 vs. 0.9850.985 for repeat-fill) to "the step artifact that zero fill adds at the window boundary". However, repeating a 2 s2\text{ s} segment 5 times introduces four boundary step discontinuities whereas centered zero-padding introduces only two. Rather, zero-filling 8 s8\text{ s} of a 10 s10\text{ s} window zeroes out 8 of the 10 unpooled temporal patch slots in vec⁡(Lm(Z))\operatorname{vec}(\mathcal{L}_m(Z)) and reduces total window energy by 80%80\%—triggering the exact same sub-chance amplitude shortcut (AUROC≈0.072<0.50\text{AUROC} \approx 0.072 < 0.50) identified in S3.3!
  2. Unregularized Flattened Readout Overparameterization (Up to 337,920337{,}920 Features), Uninformative Task/Model Pairs, and Coupled Checkpoint Selection:
    • Flattened vs. Pooled Readout Confound (Table A.3): Flattening (P,C,dh)(P, C, d_h) without temporal or spatial pooling creates 36,00036{,}000 to 337,920337{,}920 readout inputs per window (116k–1.35M116\text{k}\text{--}1.35\text{M} parameters on 6k–8k6\text{k}\text{--}8\text{k} training windows for Motor/Emotion), which inherently forces the readout weights Wm,tW_{m,t} to index rigid (patch,channel)(\text{patch}, \text{channel}) coordinates rather than translation-invariant pooled features.
    • Floor Effects & BIOT Montage Drop (Tables A.2, C.3, C.6): All 5 models are near chance on SEED-IV Emotion (AUROC 0.497–0.568\text{AUROC } 0.497\text{--}0.568, L0≤0.068L_0 \le 0.068), and BIOT is at chance on Motor (0.5110.511) and Emotion (0.4970.497) while its fixed 18-channel bipolar adapter leaves occipital channels (O1/O2) completely unmapped on 6-channel ISRUC Sleep (making BIOT's 0.0000.000 occipital sensitivity in Table C.6 an artifact of disconnected channels).
    • Coupled Checkpoint Rule (Eq. 3): Even though the backbone is frozen and the 4 task heads share zero parameters, Stage I selects one shared epoch e⋆e^\star across all 4 tasks using mean validation Balanced Accuracy—even for the 3 tasks (Motor, Seizure, Emotion) evaluated with AUROC.
  3. Undisclosed Synthetic Calibration Gate Failures in Main Text (Table E.3), Missing Clean-EEG Baseline in S3.4, and Mathematical/Statistical Inconsistencies (Tables B.6, B.7):
    • Undisclosed Gate G1 Failures: Section 4.1 does not disclose that in Table E.3 (p. 40), BIOT and LaBraM failed the Gate G1 observability threshold (<0.75< 0.75 AUROC) on all 3 synthetic spatial calibration tasks, and CBraMod failed Gate G1 on the spectral alpha calibration task. Also, Figure E.2a labels the 4 s4\text{ s} and 6 s6\text{ s} apertures as "17% Overlap" and "83% Overlap" (geometric duration overlap with [6.75,8.25] s[6.75, 8.25]\text{ s}), whereas Table E.5 and Section 4.1 report burst energy fractions of 0.6%0.6\% and 99.4%99.4\% without specifying the burst envelope window.
    • Missing Clean Reference in S3.4 & N=3N = 3 Subject Bootstrap in Table B.7: S3.4 (Table B.4) computes HnormH_{\text{norm}} only on non-brain noise/ECG/PPG without a clean-EEG reference, and Table B.7 computes 1,000-draw subject-level bootstrap 95% CIs over only N=3N = 3 test subjects on Seizure and Emotion (only 10 unique multisets) using unlabeled single-seed (20260528) point estimates.
    • Mathematical & Notation Issues in Table B.6: (a) Clipping Rraw(λ)=Q(λ)/Q(0)R_{\text{raw}}(\lambda) = Q(\lambda)/Q(0) at 00 instead of Qch/Q(0)Q_{\text{ch}} / Q(0) allows Δˉs=Q(0)[1−AURC]\bar{\Delta}_s = Q(0)[1 - \text{AURC}] to exceed the entire above-chance lift L0L_0 (e.g., CBraMod Seizure S2.1: Δˉ=0.453>L0=0.387\bar{\Delta} = 0.453 > L_0 = 0.387); (b) in S2.3 low-pass sufficiency, a model relying 100% on slow δ\delta (≤4 Hz\le 4\text{ Hz}) gets AURC=1.0\text{AURC} = 1.0 and ΔˉS2.3=0.000\bar{\Delta}_{\text{S2.3}} = 0.000, which has the opposite polarity of high-frequency sensitivity on the Figure 3 radar plot (and the 45 Hz45\text{ Hz} low-pass cutoff exceeds ISRUC Sleep's 35 Hz35\text{ Hz} storage filter); and (c) "retention" means Q(λ)/Q(0)Q(\lambda)/Q(0) in Table B.6 (RrawR_{\text{raw}}) but [Q(λ)−Qch]/L0[Q(\lambda) - Q_{\text{ch}}]/L_0 (RadjR_{\text{adj}}) in Appendix F.2 (Table F.2).
Questions for authors
  1. Fixing the S2.3 Reference Normalization (Table D.1, Clarification): If S2.3 normalizes each filtered channel x~c\tilde{x}_c to match its own pre-filter clean RMS ∥xcclean∥RMS\|x_c^{\text{clean}}\|_{\text{RMS}} (rather than unit RMS =1 μV= 1\,\mu\text{V}, which destroys 76–94% of CBraMod and CSBrain's lift on Sleep and Seizure before any filter is applied), how do the S2.3 low-pass sufficiency and band-stop necessity profiles change for CBraMod, CSBrain, and LaBraM?
  2. Disentangling S2.1 Zero-Fill from Window Energy and Unpooled Patch Slots (Clarification): What happens to S2.1 zero-fill retention on Seizure and Sleep if (a) the truncated zero-padded window is rescaled to preserve pre-truncation total window RMS energy, or (b) the linear readout is applied after temporal mean-pooling over the active (non-padded) patches rather than flattening all PP patch slots?
  3. Synthetic Calibration Gate Failures (Table E.3, Clarification): Why did BIOT and LaBraM fail Gate G1 (<0.75< 0.75 AUROC) on the synthetic planted spatial tasks (Table E.3) and CBraMod fail Gate G1 on the synthetic alpha task, and shouldn't these calibration exclusions be explicitly reported in Section 4.1?
  4. Clean-EEG Reference for S3.4 (Clarification): What is the clean-EEG covariance-spectrum entropy Hnorm(clean EEG)H_{\text{norm}}(\text{clean EEG}) for each of the 5 models across the test splits, and how does it compare to the thermal-noise, ECG, and PPG values in Table C.15?
6 comments
AuthorsOpenReview ↗

Response to Reviewer ahiv (1/4): Q1

Thank you for the detailed review and for recognizing the value of paired controls and closed-loop readout refits. Your concerns identified important confounds in the S2.3 spectral reference, the interpretation of S2.1 zero-fill responses, and the S3.4 entropy comparison. We performed the requested self-RMS correction, RMS-restoration and active-patch pooling controls, synthetic-gate follow-ups, and clean-EEG entropy comparison. These tests led us to revise several interpretations, including the physiological Sleep δ reading and the limits of entropy-based rejection. We address Q1–Q4 in order, followed by the remaining W2–W3 concerns. Tables I.x are in Appendix I of the Revised Manuscript; changes are marked in blue.

Line numbers (L) refer to the Original Submission; "now" refers to the Revised Manuscript.

Q1. How do the S2.3 low-pass and band-stop profiles change after matching each channel’s pre-filter RMS?

Self-RMS matching changes the Sleep and Seizure readings; the qualitative weak μ/β response on Motor persists. All five Sleep routes show δ-band sensitivity under the corrected reference.

Each filtered channel is now rescaled to its own pre-filter RMS, with a matched full-spectrum reference (Tables I.25–I.27), replacing the unit-RMS reference whose cost was reported in Table D.1 (L1761–1768). This measures spectral sensitivity at preserved channel amplitude, not the uncompensated energy loss from filtering.

Sleep. The full-spectrum reference of CBraMod, CSBrain and LaBraM rises from 0.244, 0.259 and 0.396 to 0.413, 0.555 and 0.507. The δ band-stop then removes 0.41–0.52 of the matched reference for those three, with substantial δ effects also present for BIOT and REVE (0.632 → 0.209 and 0.579 → 0.230; §4.3, L366–368; Table B.5, L1010); all five routes thus show the physiologically expected δ sensitivity. The three fixed-scale routes also show β sensitivity (0.16–0.18), while their σ collapse is at most 0.075 (Tables I.25, I.28).

Table I.25 — Sleep: unit-RMS → self-RMS matching, same checkpoint. Collapse values are fractions of each corresponding reference score, not absolute balanced-accuracy losses.

ModelFull-spectrum referenceLowest low-pass retaining 90%δ collapseβ collapseσ collapse
CBraMod0.244 → 0.41330 → 30 Hz0.033 → 0.4120.075 → 0.156−0.020 → −0.029
CSBrain0.259 → 0.55511 → 30 Hz0.308 → 0.5190.025 → 0.1770.013 → 0.075
LaBraM0.396 → 0.50730 → 30 Hz0.429 → 0.4430.128 → 0.1680.007 → 0.072

Seizure. Under matched RMS, a 4 Hz low-pass retains about 97% of the reference on all three fixed-scale routes, and their largest individual tested band-stop collapses are 0.027, 0.041 and 0.077 (Table I.26). The earlier apparent need for high-frequency content was therefore sensitive to reference normalization. This does not imply that frequency content is irrelevant: bands may be redundant and RMS restoration changes the scale of surviving content. The result is consistent with S3.3's amplitude-statistic evidence. On Motor, the qualitative weak μ/β response (below 2% in §4.3, L368–369) persists, although full-spectrum scores change (Table I.27). These are single-checkpoint controls, not new three-seed estimates.

Revised §4.3 accordingly states δ as an expected physiological Sleep dependency, β as an additional Sleep sensitivity, and all S2.3 conclusions as conditional on the matched-RMS reference.

AuthorsOpenReview ↗

Response to Reviewer ahiv (2/4): Q2

Q2. What happens to S2.1 zero-fill retention after RMS restoration or active-patch pooling?

The new controls identify substantial energy and padded-patch aggregation effects, without uniquely locating every residual loss in the backbone.

Completion policy. With the same visible segment, repeat fill (the paired control of §4.3, L359–363) keeps 95–99.5% of the score and zero fill 50–91%. We withdraw the original attribution to a unique boundary-step artifact (L360): repeat filling also introduces discontinuities. The result establishes completion-policy sensitivity, not stationary information sufficiency or an isolated backbone defect.

Seizure. In CBraMod, restoring window RMS raises mean zero-fill retention from 50.0% to 67.4%; in a separately fitted time-averaging head, excluding padded patches raises it from 52.6% to 92.7% (repeat fill: 98.7%). CSBrain shows the same direction: RMS restoration 90.7%→96.9%, pooled all→active patches 89.0%→97.5% (Table I.29). These are averages across aperture lengths, not individual 2 s AUROCs.

Sleep. Restoring RMS changes CSBrain retention little (60.2%→58.9%). A separately fitted time-average readout gives 81.7% with all patches and 81.6% with only active patches, versus 97.0% under repeat fill with that readout (Table I.29). The remaining difference may involve contextualized representations and other input-processing effects; it does not uniquely establish a backbone defect.

Table I.29 — Retention averaged across aperture lengths (% of each route's clean score). The pooling head is separately trained; its clean reference differs from the original flattened readout.

Task / modelOriginal zero fillRMS restoredPooled all → active patchesRepeat fill, pooled route
Seizure / CBraMod50.067.452.6 → 92.798.7
Seizure / CSBrain90.796.989.0 → 97.598.2
Sleep / CSBrain60.258.981.7 → 81.697.0

Because temporal pooling requires a separately refitted readout, each comparison uses that route's own clean reference. Recovery implicates interface and aggregation effects; a residual gap does not isolate the backbone as the sole cause. The practical implication is to audit padding and aggregation for the intended route (§4.3; §5.1, L435–437).

  • W2, flattened readout: we agree that it ties weights to patch/channel positions. With channel positions retained but temporal patches averaged, the Motor swap loss changes from 0.280 to 0.183 AUROC for CBraMod and from 0.272 to 0.236 for CSBrain; Seizure zero-fill retention changes modestly (50.0%→52.6% and 90.7%→89.0%; Table I.5). Pooling over both channels and time yields near-chance Motor AUROC (0.513–0.520; Table I.4). These comparisons implicate spatially indexed features, rather than readout size alone.
AuthorsOpenReview ↗

Response to Reviewer ahiv (3/4): Q3

Q3. Which models fail Gate G1, and what do the follow-up tests reveal about these failures?

Section 4.1 now names excluded routes. Additional controls suggest possible causes, but do not uniquely identify each failure mechanism.

Gate G1 first tests whether a route learned the planted task (AUROC ≥ 0.75; Appendix E, L1963–1964); only eligible routes can establish whether the perturbation recovers its planted dependency. Revised §4.1 and Table E.3 (already in Appendix E.1, L2023–2033) report three eligible routes for spatial calibration, four for spectral alpha, and five for temporal aperture. The new S1.2/S3.3 planted tasks have separate eligibility conditions (Tables I.30–I.31). Thus five protocols have conditional calibration, not all eight across every model. The synthetic tasks are learned by other models on the same data (alpha 0.97–1.00, spatial 0.79–0.87; Table E.3; Table I.32).

To probe sensitivity to the synthetic carrier scale, we varied the declared sampling rate and refitted the readout without changing the channel map (fault condition F3 in Table E.2, L2003–2008; Table I.32). This changes the time/frequency interpretation of the input and tests a possible explanation, not a unique cause.

BIOT stays below the spatial gate (0.711→0.725), consistent with dilution under its global-pooling interface ((1, 1, 256) in Table A.3, L814–818). LaBraM improves from 0.735 to 0.926 under the altered declared sampling rate, indicating sensitivity to the temporal/frequency scale rather than uniquely proving a timing defect. CBraMod stays below the alpha gate (0.659→0.678), and its cause remains unresolved (Table I.32).

These failures limit the models on which each protocol was calibrated; they are not evidence of a unique architectural failure mechanism. We therefore avoid assigning BIOT's result solely to pooling or LaBraM's solely to timing, while CBraMod's alpha failure remains unresolved.

  • W3, Figure E.2a and Table E.5: the 17%/83% labels measure duration overlap, whereas 0.6%/99.4% are burst-energy fractions. The envelope concentrates energy near the center; Appendix E.3 now states its exact definition (a Hann envelope centered at 7.5 s with half-width 0.75 s).
Reviewer_bQzFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a diagnostic benchmark that tests which signal properties EEG foundation models rely on, instead of comparing task scores. Five released models (BIOT, CBraMod, CSBrain, LaBraM, REVE) are each fixed with a frozen backbone and a linear readout, trained on four single-dataset tasks: motor imagery, sleep staging, seizure detection, and emotion recognition. Eight input perturbations are then applied without retraining, most paired with controls for side effects, and three are calibrated on synthetic EEG with planted dependencies. The main claim is that predictions rely on coarse properties rather than physiologically expected fine structure. On motor imagery, swapping hemispheres costs 0.18–0.28 AUROC while masking the central region costs under 0.04. On seizure detection, two models still detect seizures when the signal is replaced by amplitude-matched noise. The paper also shows that retraining only the readout with mirrored examples makes predictions respond more consistently to hemisphere swaps, and argues that some behaviors can be fixed downstream while others are targets for pretraining.

Critical strengths

The main contribution I see is methodological. The paper recognizes that a drop in score under an input perturbation can reflect side effects of the operation itself (fewer channels, padding at the window boundary, added energy) rather than the dependency being tested, and it pairs most protocols with a control designed to isolate that effect. These controls change the interpretation in several cases. A substantial fraction of the large regional losses are matched by equal-channel scattered masking (66 of 206, Appendix C.2), and most of CBraMod’s seizure truncation loss disappears under repeat fill. The calibration on synthetic EEG with planted dependencies and pre-declared fault conditions (Section 4.1, Appendix E) also checks something that random-label baselines cannot: for S1.1, S2.1 and S2.3, the readings recover the planted structure and shift as expected when the channel map or sampling rate is wrong. I think this framework of controlled perturbations plus calibration could be useful to others evaluating EEG models, independent of the conclusions drawn here.

Critical weaknesses

I consider a diagnostic benchmark a valuable contribution in principle, but its value depends on whether its readings are valid and whether they generalize, and I do not think the current experiments establish either.

On validity, the central claim that predictions rely on coarse structure rather than physiology depends on a classification of findings that conflicts with the paper’s own priors. Hand motor imagery is contralateral, so a model that uses it should reverse its left/right predictions under a hemispheric swap. A drop of about the full lift (AD 0.18–0.28 against L0 0.18–0.26, Section 4.3) is therefore the expected outcome, and Appendix D.1 concedes that the reading is not a lateralization mechanism. Likewise, δ is a pre-specified Sleep target in Table B.5 and defines N3, yet Section 4.3 counts δ reliance as weak use of physiological rhythms. The spatial masking results also cannot separate unused information from redundant information. The central region excludes the FC rows (Appendix C.2), and volume conduction spreads sensorimotor activity to neighboring channels, while the planted calibration (Table E.1) confines the signal to the target channels and cannot detect this case. On Motor, eye movements are a plausible confound, since EEGMMIDB cues appear at different screen positions per class. Finally, the two findings highlighted in the abstract come from protocols that were not calibrated (S1.2 and S3.3). For S3.3, scaling the input by 0.1 takes fixed-scale models far outside their training range, and the below-chance AUROCs this produces (Table C.5) look more like failure out of distribution than a preference for amplitude over waveform.

On generality, the evidence is narrower than the claims. Each model is evaluated in one frozen linear-probe state only, clean performance is weak (Emotion lift at most about 0.07, BIOT without lift on Motor), and Seizure and Emotion have three test subjects each. With three subjects, the bootstrap intervals in Table B.7 are effectively ranges, and the seed variance reflects only readout training. Readout sizes also differ by three orders of magnitude across models (Table A.3). More importantly, each task uses a single dataset and there are no reference routes outside the foundation models, so it is not possible to tell whether the profiles reflect pretraining or the four datasets. I would like to see the protocols applied to randomly initialized backbones with the same readout, a supervised baseline, and simple features such as per-channel log-RMS, which may well explain the seizure result on its own. I would also encourage the authors to consider running their protocols on the NeuroAtlas benchmark (Kontras et al., 2026), which the paper already cites. It provides several datasets per task and larger clinical cohorts under a unified frozen-encoder probe, and it already includes supervised and time-series foundation model baselines. This would show whether the profiles hold across cohorts and whether they are specific to EEG-FMs. As it stands, I do not think the evidence supports a recurring pattern across five models and four tasks.

The proposed path from diagnosis to improvement is also limited. The mirror refit corresponds to existing channel-reflection augmentation (Wang et al., 2024; Rommel et al., 2022), and its gain in Section 5.1 is reported against the shipped readout rather than the equal-budget control. The pretraining targets in Section 5.2 remain hypotheses. The paper should also discuss prior work on calibrating attributions with synthetic ground truth (Ismail et al., 2020; Wilming et al., 2022).

A few smaller points. Section 4.3 uses an operator-dependent SCS (CSBrain Sleep frontal, 0.030 under zeroing against 0.177 under shuffling, Appendix C.2), contrary to the rule in Section 3.3. The S1.1 scorecard region is selected on one seed without a stated criterion. Table B.7 appears to report a single seed while the scorecards report three. The main text depends heavily on custom terminology and on the appendix.

Questions for authors
  1. How do you decide, before seeing results, which dependencies count as physiological? Contralateral motor activity predicts that left/right predictions reverse under the hemispheric swap, and δ is a listed Sleep target in Table B.5. What outcome would you have read as physiological in these cases?

  2. Do the profiles change for routes that are not foundation models, e.g., randomly initialized backbones with the same readout, or a logistic regression on per-channel log-RMS?

  3. How was the S1.1 scorecard region chosen? And which spatial claims in Section 4.3 still hold if you keep only cells where zeroing and shuffling agree (cf. CSBrain Sleep frontal, 0.030 vs 0.177)?

  4. Have you checked S1.2 and S3.3 on planted synthetic tasks, as you did for the other three protocols? For S3.3, is the 0.1 attenuation within the amplitude range the fixed-scale models saw in training?

  5. Which PhysioMI runs are used? Since cue position on screen differs by class, can you rule out eye movements, e.g., by masking only Fp/AF channels?

7 comments
AuthorsOpenReview ↗

Response to Reviewer bQzF (1/4): Q1

Thank you for the careful review. Your observations that hemispheric reversal on Motor and δ-band sensitivity on Sleep are consistent with physiological priors led us to revise their interpretation in §4.3. Your concerns about validity and generality also motivated additional spatial controls, reference routes without EEG pretraining, planted calibration for S1.2/S3.3, and a second Motor cohort. We further tested nonlinear and fine-tuned states and corrected the fixed-scale S2.3 reference to self-RMS matching. We answer Q1–Q5 in order and map your Critical Weaknesses as W1 ("On validity..."), W2 ("On generality..."), W3 ("The proposed path..."), and W4 ("A few smaller points..."), addressing W1/W2/W4 inline and W3 at the end. Tables I.x are in Appendix I of the Revised Manuscript; changes are marked in blue.

Line numbers (L) refer to the Original Submission; "now" refers to the Revised Manuscript.

Q1. Which dependencies count as physiological?

Both the left–right class-ranking reversal under hemispheric swapping in Motor and the δ-band sensitivity in Sleep are consistent with physiological priors. The former does not localize a cortical source or establish μ/β reliance; the latter supports δ dependence but does not establish reliance on other bands, such as σ.

The task-related targets and diagnostic controls were specified in Table B.5, fixed before acquisition (L1004–1005); Table I.17 clarifies the hierarchy used to interpret them. We do not claim that a complete physiological/nonphysiological classification was preregistered.

Motor. Scored per class, the original swap drop (0.18–0.28 AUROC; L350–354) shows that the left and right predictions mirror under the swap, as you predict. The AUROC of the left and right classes falls to 0.23–0.38 and, scored against left↔right-exchanged labels, returns to the original value in all four models with lift. The earlier argmax-reversal measure (L355–357) has a state-dependent ceiling of 0.60–0.72 (Appendix F.1, L2278–2282) and must be distinguished from these class-wise rankings; §4.3 now reports both.

Left / right AUROC (Table I.18)OriginalSwappedRelabelled
CBraMod0.795 / 0.7820.231 / 0.2360.800 / 0.789
CSBrain0.820 / 0.7850.257 / 0.2330.806 / 0.797
LaBraM0.731 / 0.7140.377 / 0.3570.736 / 0.711
REVE0.811 / 0.7800.350 / 0.3270.790 / 0.773

The mirror shows lateralized class-ranking correspondence, not a unique cortical source. Central sensorimotor content and μ/β rhythms are more specific targets: their use can yield mirroring, but mirroring alone does not establish their necessity.

Frequency. Removing δ, the pre-specified Sleep target (Table B.5; §4.3, L365–368), costs every model 0.41–0.67 of its reference score, whereas the narrower targets cost little: μ and β on Motor at most 0.02 (below 2% in L368–369), and the spindle band σ on Sleep at most 0.075 in four models (Table I.28). δ also costs 0.08–0.19 on Motor, where it is not a target. §4.3 now names the level that is used and the levels that are read weakly.

  • W1 (Validity), unused versus redundant: as you point out, masking cannot separate the two, and the swap exchanges electrode positions without moving content, so we tested the cortical level separately. If left–right information were confined to the central region, removing it would abolish the mirror; instead, the mirror survives with the 14 central channels zeroed (Table I.20), consistent with spatial redundancy (including volume conduction, already named in L144–145). Thus the left/right correspondence is not exclusively dependent on the predefined central subset, nor abolished by anterior ablation; neither establishes neural-source localization.
  • W1 (Validity), planted calibration: the original planted task confines the label to target channels and does not validate the spatially redundant case. The real-data masking-before-swap test checks whether the correspondence survives without these channels; it does not isolate the underlying redundancy mechanism.
  • W1 (Validity), FC row: adding seven FC electrodes leaves the regional loss below the matched scattered control except for a negligible 0.002 AUROC excess for LaBraM under zeroing (not under shuffling; Table I.19). Thus the central-masking finding (L321–323) is not explained solely by the original atlas boundary.
AuthorsOpenReview ↗

Response to Reviewer bQzF (2/4): Q2

Q2. Are the findings specific to EEG pretraining, readout states, or datasets?

The measured effects persist outside EEG-pretrained models, changing their interpretation: neither response is specific to EEG pretraining.

Seizure. Your conjecture about the replacement finding (L391–392) is correct: a logistic regression on per-channel log-RMS reaches AUROC 0.883 on Siena, against 0.884 for the same CBraMod checkpoint (Table I.13), and its features are unchanged by the noise replacement by construction. A randomly initialized CBraMod and EEGNet also score at least as high on the RMS-matched noise as on the recording. This holds for Seizure only: on the other three tasks the same regression gains at most 0.10 over chance, against 0.38 on Seizure (Table I.14).

Siena, AUROC (Table I.13)RecordingWaveform replaced by RMS-matched noise
CBraMod, pretrained0.8840.896
CBraMod, randomly initialized0.7520.771
EEGNet, trained from scratch0.8550.878
Logistic regression on per-channel log-RMS0.8830.883

Motor. The predictions also mirror under the swap with the randomly initialized CBraMod and with MOMENT, a general time-series foundation model: the swap removes 0.186 AUROC at a lift over chance of 0.153, and 0.285 at a lift of 0.232 (Table I.15). Over the random backbone, pretraining adds 0.13 AUROC on Seizure and 0.10 on Motor. The revision no longer attributes these readings to EEG pretraining and reports the reference routes beside the foundation models.

  • W2 (Generality), second cohort: we added BCI Competition IV-2a as an initial second cohort, where the mirror reappears in three of the four models (Table I.16). CSBrain's left and right classes are only 0.08–0.11 above chance there, so the swap does not take them across chance. The abstract's claim is stated per task and cohort.
  • W2 (Generality), NeuroAtlas: To address your NeuroAtlas suggestion, we evaluated two components also covered by that benchmark: MOMENT-1-small, a general time-series foundation model (Table I.15), and BCI Competition IV-2a, an additional Motor cohort (Table I.16). We applied our own fixed-route diagnostic protocols rather than reproducing the full NeuroAtlas evaluation. The hemispheric-swap response appears in MOMENT and in three of four evaluated models on IV-2a, providing evidence beyond EEG-specific pretraining and partial cross-cohort replication. These results do not establish universal generality; broader evaluation across NeuroAtlas cohorts remains an important extension.
  • W2 (Generality), states: each profile belongs to one state (adapter, training regime and readout), and the revision states every finding for the state in which it was measured. We added full fine-tuning and a nonlinear head on the frozen backbone (declared in Table A.5, L874–885). Under full fine-tuning Motor AUROC rises by 0.06–0.16, and removing μ and β still costs at most 0.03 (Table I.3); on the original three-subject Seizure test, the fully fine-tuned CBraMod and CSBrain retain AUROCs of 0.873 and 0.869 after replacement (Table I.8). Added-state results are principally single-seed evaluations.
  • W2 (Generality), subjects: in five-fold subject-disjoint cross-validation over all 14 Siena subjects, CBraMod and CSBrain retain 67% and 78% of clean lift after full replacement (Table I.8). Their replaced AUROCs are lower than clean (0.812→0.709 and 0.919→0.827), unlike the original three-subject comparison; this strengthens within-cohort evidence, not external generalization.
  • W2 (Generality), readout size: in CBraMod the swap removes 0.18, 0.28 and 0.27 AUROC with 51,204, 204,804 and 3.3 million readout parameters (Table I.4), so the reading does not follow the size of the readout. Pooling away spatial indexing also removes most Motor lift in the tested heads, indicating that retained channel-resolved features matter for this evaluation route.
  • W2 (Generality), cells without lift: the canonical frozen routes have insufficient baseline observability, the case that Step 1 of §3.3 sets aside (L200–202). Task-specific Emotion readouts do not materially improve test AUROC; full fine-tuning brings three models to about 0.60 (Table I.10). This does not uniquely identify a backbone defect. BIOT's low Motor lift is consistent with its globally pooled feature interface (Table A.3, L812–818), but cannot isolate the pooling mechanism from other route differences.
AuthorsOpenReview ↗

Response to Reviewer bQzF (3/4): Q3–Q4

Q3. How is the displayed region selected, and do zeroing and shuffling agree?

Across 60 readable spatial cells, 44 have operator-consistent interpretations, including all 20 Motor cells; consistency is not equivalent to positive regional specificity.

The audit applies the operator rule of §3.3 (L203–205) and identifies 44/60 operator-consistent cells (Table I.24). On Motor the central region stays within its scattered control under both operators, and the frontal region exceeds its control under both in CBraMod and CSBrain. The two Sleep frontal-specificity claims—CSBrain (§4.3) and REVE (Appendix C)—have been withdrawn. One Motor statement is narrowed: frontal masking costs CSBrain almost twice as much as CBraMod under zeroing only.

The scorecard descriptively selects the region with the largest zeroing-based area under the relative-loss curve on the reference seed; this is not a prespecified physiological-target test. Under the zeroing operator, the three seeds select the same region in 13 of 14 readable model–task cells, and on Motor all four models select the frontal region under every seed and operator (Table I.23).

  • W4 (Smaller points): Table B.7 comes from a single fitted state (seed 20260528), which Appendix B.7 now states; in §4 the custom abbreviations are replaced by what each quantity measures.

Q4. Are S1.2 and S3.3 calibrated, and how is attenuation handled?

The additional synthetic tests support selected readings conditional on task learnability; RMS matching does not guarantee an in-distribution waveform.

Swap. Extending the planted-task calibration of §4.1 (L253–259; Appendix E), we planted the label in the two sides in opposite directions, in one side only, or in both sides equally. A flip index (1 for a full reversal, 0.5 for a fall to chance, 0 for no change) gives 0.96–0.99, 0.32–0.46 and 0.00 ± 0.01 in the models that learn the tasks. The real Motor flip indices (0.74–0.97) more closely resemble the planted bilateral-reflection condition than the unilateral or symmetric controls (Table I.30); they cannot uniquely identify a neural source.

Crossfade. The waveform is replaced by pink noise at each channel's original RMS; this preserves amplitude statistics but does not guarantee membership in the full EEG training distribution. On the planted task whose label is in the waveform, this takes the three models that can be read to chance (0.42–0.52), whereas on the planted task whose label is a gain CSBrain keeps its score (0.896 → 0.886); these tests distinguish waveform-coded from gain-coded labels in the eligible synthetic settings (Table I.31). Calibration depends on synthetic-task learnability: CBraMod is not readable in these S3.3 planted tests, and LaBraM gain-coded AUROC falls from 0.831 to 0.657 after replacement, which limits the reading of a score drop more than that of a retained score. Thus this is conditional operator calibration, not validation of every model-specific real-data interpretation. The real-data result more narrowly shows that substantial discrimination survives after removing the original waveform; it does not prove an EEG-FM-specific or nonphysiological shortcut.

0.1 attenuation. At this setting (L393–395) 89% of the Siena test windows are quieter than the first percentile of the windows the readouts were trained on (Table I.9). On the planted tasks it takes a model below chance only where the label is carried by gain, so out-of-range input and reliance on amplitude cannot be told apart. The abstract now states the amplitude finding from full replacement only, a test the Original Submission already reported (L391–392).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the Area Chair and Reviewers: Core Highlights and Revision Overview

We thank the Area Chair and reviewers for two constructive discussion rounds. In the revised manuscript (marked in blue; 40 new tables in Appendix I, §I.1–§I.9), we aligned our core claims from pretrained EEG-FMs and EEG pretraining to downstream task scores along specific evaluation routes (adapter, backbone, readout) and cohorts. Input perturbations test which signal cues sustain a route's score rather than proving what a backbone fails to encode or attributing effects to EEG pretraining.

Work Highlights

  1. Controlled, calibrated diagnostic methodology that decouples adapter, backbone, and readout before raw-input probing, pairs perturbations with side-effect controls, and conditionally validates five protocols on synthetic planted ground truth (a methodological design recognized by all three reviewers).
  2. Score-level signal diagnosis showing across five released models that evaluated route scores can be sustained by coarse signal cues, while narrower physiological targets are not necessary under the tested interventions (whether unused or redundant).
  3. Actionable interface-vs.-backbone attribution via frozen-backbone readout refits against equal-budget controls, separating behaviors that can be modified at the downstream interface (§5.1) from falsifiable hypotheses for future pretraining (§5.2).

Summary of Revisions

Focus (Reviewers)What We Added / UpdatedOutcome for Claims
States, cohorts & baselines (FrXa Q1–Q3, bQzF Q2/Q5/FU)Nonlinear & single-seed fine-tuned states, 14-subj. Siena CV, BCI IV-2a (+ EOG regression), non-EEG baselines (§I.1–§I.4, §I.9), and initial single-seed HMC/Helsinki checks (bQzF FU).Motor and Siena-CV lift-retention patterns persist beyond frozen linear probes and 3-subject splits; route and cohort boundaries (including Siena-specific replacement retention) are separated from EEG pretraining.
Controls & calibration (ahiv Q1–Q4, bQzF Q1/Q3–Q4/FU)Self-RMS S2.3 reference, energy/active-patch S2.1 controls, S1.2/S3.3 & spread/redundancy calibration (5 protocols total), Gate G1 & clean S3.4 reference, 31-ch expansion, agreement & 0.1× audits (§I.2, §I.5–§I.9).Recovers expected Sleep δ/β and Seizure 4 Hz retention, withdraws inconsistent Sleep frontal and OOD 0.1× claims, and supports the paired-control calibration design.
Framing & presentation (ahiv FU, bQzF FU, FrXa W4)Score-centered non-necessity wording, falsifiable §5.2 hypotheses, lift/flip split (Fig. 3, Table B.6), RrawR_{\mathrm{raw}}/RadjR_{\mathrm{adj}} naming, LaBraM G1 note (§4.1), and plain main-text terms (§1, §3–§5, Table I.38).Shifts claims from EEG-FM "weak use" or pretraining defects to evaluated route scores (unused or redundant finer targets) with clearer, consistent notation.

These reviewer-guided revisions align our claims with the controlled evidence and strengthen the benchmark's rigor. Full multi-cohort Sleep/Seizure expansion is left for the next version alongside our initial HMC/Helsinki checks, and the locked OpenReview TL;DR will be aligned once editable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.