- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
In this work, the authors compared different EEG FMs with their randomly initialized versions and tried to isolate the contribution of self-supervised pretraining across diverse downstream tasks. They find that the gain is task-dependent. Then, they investigate which signal properties support pretraining versus downstream decoding. Their results suggest that current reconstruction- and prediction-based pretraining objectives can rely heavily on spectral and cross-spectral information, while failing to reliably acquire or retain temporal/task-relevant representations needed for MI and ERP decoding.
Critical strengths
- It is well motivated to investigate and diagnose the pretraining’s contribution, and the authors have found interesting results regarding the mismatch between the pretraining task and downstream tasks.
- The experimental design is generally careful and well controlled for the specific hypotheses being tested.
Critical weaknesses
-
I appreciate the efforts and the amount of work the authors have put into the benchmark. As stated in the abstract, the broad benchmark covering 23 encoders and 34 datasets can convincingly establish task-dependent pretraining gains. However, the paper’s main mechanistic conclusions are not derived from this large-scale benchmark, but from a much narrower set of controlled experiments centered primarily on REVE and a small subset of datasets. It therefore remains unclear whether the results and mechanistic observations generalize across diverse architectures, particularly models with substantially different tokenization schemes, pretraining objectives, and sizes. In this sense, the headline emphasis in the abstract on “23 encoders and 34 datasets” may overstate the breadth of evidence supporting the paper’s mechanistic claims, since the large-scale benchmark mainly establishes the descriptive phenomenon, while the follow-up diagnosis is validated in a substantially narrower setting.
-
Given that another central motivation for foundation models is scaling, this study would benefit from explicitly analyzing whether the observed objective–task mismatch changes with model size, pretraining data, or compute. The current REVE-small/base and 5×-budget experiments partially answer this, but I find it hard to conclude that scaling cannot recover the missing MI/ERP information.
-
The visualization and presentation of the results could be improved. Given the rich results from the benchmark and the large number of carefully controlled experiments with paired data, I believe the paper could extract and present considerably more insight from the collected results. For example, it would be informative to examine how pretraining gain varies with encoder scale, model capacity, pretraining data size, or compute, and whether these relationships differ across task families. Also, the contribution of different preserved signal properties could be summarized more systematically across downstream tasks rather than being presented primarily through a limited set of aggregate comparisons. Even the current main figures could be made more informative. In Figure 1, the pretrained-versus-random gain is useful for isolating the contribution of pretraining, but presenting an absolute supervised reference (e.g., EEGNet) alongside it would help distinguish between tasks where pretraining adds little because the task is already well captured by simple representations and more challenging tasks. In Figure 2, the paired nature of the comparisons could be better exploited by reporting effect sizes and confidence intervals, or formal significance/equivalence analyses, directly in the main figure.
-
Longer reconstruction training does not necessarily guarantee that all task-relevant features have been learned. Reconstruction may first capture dominant, easier-to-model low-frequency structure, while weaker MI-relevant mu/beta dynamics or temporally localized patterns may emerge only later. Tracking frequency- and time-specific representation quality throughout pretraining would therefore help rule out the possibility that the current models are simply undertrained or underfitting the reconstruction objective. As a useful limiting thought experiment, if a masked-reconstruction model could perfectly reconstruct the full EEG signal from its visible context, then the encoder–decoder system would necessarily have captured essentially all predictable signal structure, including information relevant to MI and ERP. From this perspective, the weak downstream transfer observed here could reflect not only the objective–task misalignment, but also the fact that current models are still far from this reconstruction limit. The manuscript would benefit from distinguishing these two possibilities.
-
The paper diagnoses the problem well but provides limited guidance on how it might be addressed. Although developing a complete solution may be outside the scope of the study, I would expect a more detailed discussion of possible self-supervised objectives that could better preserve temporal or task-relevant information for MI and ERP. In particular, the fact that the same failure persists under the tested JEPA objective suggests that the issue may extend beyond masked reconstruction, and the manuscript could discuss more explicitly what this implies for the design of future EEG pretraining objectives.
Questions for authors
Regarding Critical Weaknesses (CW) 1:
The main mechanistic analyses are conducted in a much narrower setting, primarily using REVE and six downstream datasets. This raises several important questions. To what extent do the signal-transformation findings generalize beyond REVE? Do models with substantially different tokenization strategies or self-supervised objectives (for example, contrastive approaches rather than MAE- or JEPA-style prediction) show the same dependence on spectral and cross-spectral information? Similarly, does the acquisition/retention behavior observed in Figure 3 also hold for other EEG foundation-model architectures? These questions are important because the paper’s broad benchmark establishes that pretraining gains are task-dependent, but the proposed explanation for why those gains differ is validated much more narrowly. I also suggest having the abstract more clearly distinguish the evidence from the broad benchmark from the evidence supporting the mechanistic conclusions.
Regarding CW2:
- Have the authors compared generic/random 4-s pretraining windows with event- or cue-aligned windows?
The current experiments use a common 4-s windowing scheme. However, for ERP and, to some extent, MI, much of the discriminative structure is defined relative to an event or cue. If these recordings are segmented without preserving a consistent temporal reference, task-relevant dynamics may be diluted across windows, especially when most segments in the pretraining data pool are not time-locked ERPs. It would therefore be informative to compare SSL pretraining on generic 4-s segments with pretraining on stimulus-/cue-aligned windows while keeping the architecture, objective, and amount of data fixed.
- Following question 1, this study shows that reconstruction on the same downstream corpus does not recover the supervised gain, but this does not fully answer whether the type of data and the structure of the unlabeled data matter. One important distinction is between using “more MI/ERP recordings” and using many unlabeled epochs that repeatedly preserve the same event-relative features. Therefore, how confidently can the weak MI/ERP transfer be attributed to objective–task mismatch rather than insufficient exposure to temporally aligned task structure? The authors interpret the Section 4 results as evidence that the objective is a key bottleneck because the same architecture and recordings can learn strong MI/ERP representations once supervision is added. Could a large collection of task-structured but still unlabeled MI/ERP epochs allow the same SSL objective to learn temporal information that is currently missing?
Given the scope of the comments above, I recognize that there are quite a few questions and do not expect the authors to address all of them. I would prioritize improving the result presentation and visualization (CW3) and making fuller use of the existing broad benchmark through additional analyses (CW1). Further experiments addressing the generalizability concern in CW1 would be a valuable addition, but even a small number of validations across different architectures or objectives would considerably strengthen the robustness and broader relevance of the mechanistic conclusions.
7 comments
Response to Reviewer Mg2C (2/4): CW3
CW3 — Presentation
Extract more from the existing results (gain against encoder scale, data and compute, by task family). Fig. 1: add an absolute supervised reference such as EEGNet. Fig. 2: show effect sizes and intervals, or significance / equivalence tests.
We updated both main figures and added the analyses you list.
Fig. 1 now shows absolute scores, with EEGNet as the supervised reference. Mean κ over the published encoders and datasets of each family:
| Task family | Random encoder | Pretrained encoder | EEGNet | Distance to EEGNet covered |
|---|---|---|---|---|
| Sleep | 0.27 | 0.43 | 0.53 | 63% |
| Clinical | 0.26 | 0.36 | 0.43 | 63% |
| SSVEP | 0.12 | 0.18 | 0.46 | 16% |
| Motor imagery (MI) | 0.07 | 0.10 | 0.41 | 9% |
| ERP | 0.02 | 0.02 | 0.26 | none |
MI and ERP are far from solved, so the small gain there is not a ceiling effect.
Revised Fig. 1 (Section 2.1, p. 4): absolute scores with EEGNet (a), gain from pretraining (b), gain once descriptors are given to both heads (c).
Fig. 2 now shows paired differences with 95% intervals and an equivalence test. Pretraining on PSD + cross-spectra surrogates (phase-randomized windows that keep their spectra and cross-spectra) instead of original EEG changes transfer by +0.002 κ [−0.008, +0.012]. This is equivalent to zero within ±0.02 κ, 10% of the pretraining gain (two one-sided tests, p < 0.001), and so are the other four objective, size and budget configurations. The surrogate does remove task information: it costs a supervised EEGNet 0.30 κ on MI and 0.34 on ERP.
Revised Fig. 2 (Section 3.2, p. 7): scores by signal version (a) and paired differences with 95% intervals and the ±0.02 equivalence band (b).
Also new in the appendix:
- The gain against parameters, pretraining hours and compute of the published encoders, by family (new Fig. 8, App. E.5, p. 45). No association survives correction, but with 10 to 22 encoders per panel this excludes only strong trends; the controlled runs of CW2 answer more directly (new Fig. 7, App. E.5, p. 44);
- By task, a table with the share of the pretraining gain and of EEGNet's score that each signal property keeps (App. E.4, Table 29);
- Formal statistical significance tests of the task dependence (App. B.1; 69Vo, W3): sleep and clinical gains exceed MI and ERP gains by +0.12 κ [0.07, 0.19], permutation p < 0.001.
Response to Reviewer Mg2C (3/4): CW2 and its two questions
CW2 — Scaling (App. E.5, new Fig. 7, p. 44)
Does the objective–task mismatch change with model size, pretraining data or compute?
Within the scales we could test, none closes the MI and ERP gap. We pretrained on four amounts of distinct TUEG data: 22 h, 222 h, 2,222 h and the full 23,000 h. All four runs see the same number of windows (42 M, about two epochs of the full corpus), so smaller sets are repeated more: each window of the 22 h set is seen about 2,000 times. More data stops helping after 222 h. Longer training adds little: we followed one run for 43 epochs of the full corpus (about 900 M windows), and it barely changes after the first epoch. More compute or parameters helps MI a little and ERP hardly at all, while sleep and clinical EEG already match EEGNet (six datasets of Section 3):
| Task family | REVE-small (paper) | REVE-small, trained 5× longer | REVE-base, 5.4× more parameters | EEGNet (supervised) |
|---|---|---|---|---|
| Motor imagery | 0.12 | 0.13 | 0.13 | 0.42 |
| ERP | −0.01 | 0.00 | 0.00 | 0.34 |
| SSVEP | 0.72 | 0.77 | 0.75 | 0.89 |
| Sleep | 0.54 | 0.56 | 0.55 | 0.54 |
| Clinical | 0.59 | 0.63 | 0.63 | 0.53 |
Larger scales (more than 69 M parameters, 23,000 h of data or 43 training epochs) remain untested; the conclusion now says so.
CW2-Q1 and Q2 — Cue-aligned windows, and unlabeled task epochs (App. F.7)
Q1: pretraining on generic 4-s windows vs cue- or stimulus-aligned windows, all else fixed. Q2: could many unlabeled, event-aligned MI/ERP epochs let the same objective learn what is missing?
We find window placement to make no difference, and having many more unlabeled aligned epochs barely helps. The experiments in Section 4 already consider pretraining on cue-aligned epochs: one 4-s epoch per trial, at a fixed position relative to the cue (the revised text now says so). These are about 24,000 unlabeled epochs of the target dataset, each seen about 500 times. We added the generic counterpart, with the same people, number of windows, model, objective and budget (three seeds; paired intervals contain zero):
| Pretraining windows (same corpus) | MI κ | ERP κ |
|---|---|---|
| Generic 4-s windows | 0.07 | 0.00 |
| Cue-aligned epochs | 0.06 | 0.00 |
| Cue-aligned epochs, with labels | 0.37 | 0.28 |
With 7.5 times more ERP epochs, held-out reconstruction improves but ERP decoding stays at chance; with 2.1 times more MI epochs, κ rises only from 0.065 to 0.088. The information is in the aligned epochs, and reconstruction does not extract it. A contrastive objective on the same MI epochs does better (κ = 0.22), and nearly as well (0.19) when trained on their PSD + cross-spectra versions: it too learns mostly spectra and cross-spectra (App. K).
Response to Reviewer Mg2C (1/4): CW1
Thank you for the detailed and constructive feedback.
CW1 — Beyond REVE
To address this point, we extended both analyses well beyond REVE (new App. K, new Fig. 16, p. 69). Specifically, we included all 23 published encoders of our benchmark and all 27 datasets of the five main families, instead of six. We also pretrained five new encoder–objective pairs: CBraMod (criss-cross attention over per-electrode patches) and LUNA (electrodes compressed into learned queries) with masked reconstruction, REVE-small with a contrastive objective (SimCLR), and BIOT and BENDR with their own contrastive objectives. Finally, we repeated acquisition / retention (Fig. 3, Section 4, p. 8) with CBraMod and LUNA.
The new experiments support the findings of Section 3 across architectures, tokenizations and objectives, contrastive ones included. Acquisition / retention (Section 4) holds for two other architectures.
Section 3 (pretraining on surrogates).
For the four encoders that gain from pretraining, the difference is close to zero. CBraMod's larger difference comes from one SSVEP corpus, where its attention readout is unstable across seeds. Without this corpus, the difference is −0.002. On this corpus, a readout that keeps the electrodes separate gives κ = 0.79 for both pretrainings. On all 27 datasets, original EEG keeps a small edge for these four encoders (0.007 to 0.015 κ), as in the paper's REVE runs (0.009). BIOT and BENDR do not gain from pretraining, like their published weights, so there is no gain to explain.
Section 4 (acquisition and retention, Fig. 16c–d). We repeated Fig. 3 with CBraMod and LUNA on MI, and with CBraMod also on ERP. The pattern is the same as with REVE. A small label term added to reconstruction raises frozen decoding to 0.36–0.43 κ. Reconstruction alone then erases most of this gain (at most 14% is kept), while continued supervision keeps it.
All these results are in the new Fig. 16 (App. K, p. 69 of the revised PDF): published encoders read on surrogates (a), new pretrainings (b), acquisition and retention with CBraMod and LUNA (c, d).