- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper evaluates EEG foundation models through corruption robustness, attribution maps, and frozen-feature pooling experiments. Across multiple datasets, it shows that missing-channel handling affects robustness and that preserving token-level features improves frozen-probe performance.
Critical strengths
- Practically useful robustness analysis. Comparing perturbation types and channel removal versus zero-padding exposes failure modes that clean accuracy misses.
- Substantive pooling findings. Flattening improves average frozen-feature performance for all five tested models across four tasks. Block-wise probes further demonstrate the importance of readout choice.
Critical weaknesses
- Robustness rankings require stronger controls. BrainOmni’s noise bandwidth differs from that of other models (Appendices C/D.1), so equal total SNR does not ensure matched corruption in their shared frequency range. Moreover, one dropout mask per dataset/probability is reused across folds (D.2), leaving uncertainty across missing-channel patterns unmeasured.
- The pooling explanation exceeds the controls. Mean-tile matches parameter count but not independent feature dimensions or token-specific linear weights. The results establish readout sensitivity, but do not fully separate information preservation from probe capacity and optimization. Appendix G’s end-to-end scratch models also do not isolate pre-training’s contribution to frozen features.
Questions for authors
- Does BrainOmni retain its noise advantage under a common physical corruption, with matched spectral content and amplitude before model-specific preprocessing?
- Are dropout rankings and removal-versus-padding gains stable across independently sampled masks?
- Do flattening gains persist with dimension-controlled readouts and validation-tuned regularization? How do frozen random backbones perform with the same readouts?
4 comments
Response to Reviewer GCbo (2/3)
Sampled Masks We thank the reviewer for their observation of the reused mask (this was the result of the random channel selection process). To answer the question on how robust the ranking across masked selections is, during the rebuttal period we repeated the channel droppout analysis with two more different seeds (leading to two more random masks) under two settings: (i) Zero-padding the missing electrodes and (ii) Removing the missing electrodes. The various masks are described in Appendix D.2 of the revised manuscript.
As shown in the Tables below (reporting performance in bacc mean ± std over three mask seeds), the ranking between the original reported results and under three different seeds remains largely the same in both settings (with the exception of EEGNet and BrainOmni in the zero-padding setting). These results have been updated in the revised manuscript.
Zero-Padding, p = 0.5, Full-fine Tuning
| Model | Eyes | Physionet MI | Physionet ME | KU MI | HG | Pavlov | KU ERP | Siena | Avg | Rank (seed avg) | Rank (original paper) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| REVE | -2.7 ± 3.3 | -4.1 ± 0.7 | -5.0 ± 3.0 | -5.3 ± 2.9 | -10.7 ± 3.5 | -1.0 ± 0.4 | -18.6 ± 9.8 | -20.4 ± 7.3 | -8.5 ± 1.5 | 1 | 1 |
| LaBraM | -3.3 ± 0.2 | -9.2 ± 4.3 | -11.0 ± 5.4 | -9.8 ± 2.5 | -25.3 ± 4.2 | -0.1 ± 0.2 | -19.1 ± 9.8 | -6.8 ± 0.0 | -10.6 ± 2.5 | 3 | 2 |
| EEGNet | -6.4 ± 3.6 | -5.0 ± 0.4 | -6.0 ± 1.0 | -13.9 ± 6.7 | -22.3 ± 3.2 | -0.3 ± 0.5 | -17.7 ± 8.0 | -11.0 ± 5.9 | -10.3 ± 1.8 | 2 | 3 |
| CBraMod | -3.0 ± 1.8 | -9.2 ± 2.6 | -17.2 ± 8.3 | -15.9 ± 8.6 | -23.6 ± 7.5 | -5.8 ± 2.7 | -16.9 ± 8.5 | -16.5 ± 8.7 | -13.5 ± 2.7 | 5 | 4 |
| NeuroRVQ | -7.2 ± 1.3 | -11.5 ± 1.9 | -14.7 ± 1.8 | -14.1 ± 1.6 | -34.7 ± 0.7 | -5.9 ± 1.4 | -15.9 ± 5.5 | -6.4 ± 3.8 | -13.8 ± 1.6 | 6 | 5 |
| BrainOmni | -12.2 ± 7.5 | -12.9 ± 0.4 | -12.4 ± 2.5 | -9.2 ± 0.9 | -18.6 ± 1.8 | -0.0 ± 0.2 | -19.7 ± 0.9 | -13.7 ± 4.1 | -12.3 ± 0.5 | 4 | 6 |
True Removal, p = 0.5, Full-fine Tuning
| Model | Eyes | Physionet MI | Physionet ME | KU MI | HG | Pavlov | KU ERP | Siena | Avg | Rank (seed avg) | Rank (original paper) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| LaBraM | -0.9 ± 0.2 | -1.8 ± 0.5 | -1.5 ± 0.7 | -3.3 ± 1.9 | -10.6 ± 5.4 | 0.1 ± 0.2 | -6.7 ± 5.3 | 0.9 ± 0.3 | -3.0 ± 0.9 | 1 | 1 |
| BrainOmni | -0.9 ± 1.1 | -4.5 ± 1.5 | -4.6 ± 2.0 | -2.2 ± 1.8 | -6.0 ± 1.9 | -0.3 ± 0.6 | -2.6 ± 2.0 | -3.2 ± 1.6 | -3.0 ± 1.0 | 2 | 2 |
| REVE | -4.0 ± 4.1 | -2.6 ± 1.0 | -3.3 ± 1.6 | -2.6 ± 1.1 | -8.3 ± 2.8 | -0.6 ± 0.5 | -5.5 ± 5.7 | -2.9 ± 4.5 | -3.7 ± 0.5 | 3 | 3 |
| NeuroRVQ | -1.7 ± 0.3 | -4.5 ± 1.2 | -4.7 ± 0.9 | -4.0 ± 1.3 | -16.5 ± 2.1 | -0.9 ± 0.7 | -9.0 ± 5.2 | -4.2 ± 2.9 | -5.7 ± 1.1 | 4 | 4 |
Response to Reviewer GCbo (3/3)
Flattening Gains
In the original manuscript, we demonstrate that flatten pooling head-only performance is significantly better than mean pooling performance on the same pre-trained model, demonstrating that mean pooling can discard useful task-related information. Because flattened tokens lead to an increase in the classification head, in our original manuscript we conducted an ablation study (Appendix J) with a parameter-matched control ablation, showing that token information not head size is the leading factor of this increase. Furthermore, our ablation study (frozen backbone and random from scratch analysis) in Appendix G of the revised manuscript demonstrates that pre-training improves performance on average for most models.
To examine the effect of flattening gains with dimension-controlled readouts (as suggested by the reviewer), during the rebuttal period we conducted a readout dimensionality ablation (Appendix J.1 in the revised manuscript). We compare flatten and mean readouts by performing Principal Component Analysis (PCA) to the flattened token representations, projecting them into progressively lower-dimensional spaces. We retain the top k*D (k scale factor and D the dimensionality of mean pool readout) principal components, with PCA fit on the training trials only and train an L2-regularised logistic regression on each projection. The regularisation strength is selected on a 15% validation split and we report test balanced accuracy over 10 subject-wise folds. As shown in Figure 29 in the revised manuscript (we refer the reader to Appendex J.1), performance generally improves with increasing readout dimensionality. More importantly at k=1 corresponding to the dimensionality of the mean pooling readout, all models on all tasks (with the exception of CBraMod on ERP) outperform their respective mean pooling counterparts. This finding further supports our hypothesis that mean pooling discards potentially useful task-related information, limiting head-only performance. Furthermore, we also added a new pooling strategy, namely Time-mean that averages the tokens across the time dimension for each channel individually and retains the spatial dimension. As shown in Table 23 (missing columns are currently running and will be added in the camera-ready version of the paper), time-mean consistently leads to better performance compared to mean-pooling. Our original matched-size comparison, the new dimension-controlled PCA analysis as well as the time-mean pooling result support our hypothesis that mean-pooling limits head-only performance.
We hope that these clarification, extra results and ablation studies satisfy the reviewer's concerns.
Follow-Up to Reviewer GCbo
As the discussion period is ending soon, we wanted to kindly follow up on our responses and the revised manuscript, which address the weaknesses and questions raised in your review. We would greatly appreciate it if you could let us know whether these address your concerns.
Response to Reviewer GCbo
We thank the reviewer for their constructive comments. We address every point with new evidence and clarifications.
BrainOmni Robustness Analysis. We agree with the reviewer that BrainOmni’s noise bandwidth differs from that of other models, steming from the model's original higher bandpass filtering (0.1-96Hz) compared to the default pipeline (0.5-45Hz) as stated in Appendix C. In the original paper, we demonstrate that under pink noise (1/f), a fairer comparison in the shared frequency range, BrainOmni leads in robustness (we refer the reader to Table 13 of the revised manuscript). During the rebuttal period, we repeated the robustness analysis of BrainOmni under three settings: (i) bandpass at 0.1-96Hz and original white noise (original manuscript result), (ii) bandpass at 0.1-96Hz and restricted white noise 0.5-45Hz (same as all other models) and (iii) fine-tuned on bandpass 0.5-45Hz and noise on the same range (same fine-tuning and testing as the other models but unfair to BrainOmni's original preprocessing setup). We report the drop from the clean performance for all three settings across 6 levels of noise (mean of 8 datasets, excluded the Sleep datasets as mentioned in the original manuscript) and Wilcoxon pairing test between BrainOmni and our deep learning baseline EEGNet (* denotes p<.05). The results of this analysis are reported in the table below:
As seen from the table above as well as Figure 13 in the revised manuscript (we refer the reviewer to Appendix H.1.1 for the complete analysis), BrainOmni keeps its robustness lead under matched spectral content. Therefore, our observations are fair. We thank the reviewer for pointing out this unmatched tested conditions that this new ablation study (in Appendix H.1.1 of the revised manuscript) clearly answers.