acceptodds
Under review as a conference paper at ICLR 2027

Promises and Perils of EEG Foundation Models: On the Role of Pretraining

Abstract

Foundation models for electroencephalography (EEG) typically rely on masked prediction, with the promise that pretraining on large unlabeled collections yields general-purpose representations. Yet published evaluations rarely isolate what pretraining itself contributes, leaving open whether reported gains come from representations learned from unlabeled EEG or from the encoder and adaptation protocol. We ask what pretraining contributes, whether its benefits are task-dependent, and whether the pretext task is aligned with all downstream tasks or only with some. Across 23 published encoder configurations and 34 datasets, evaluated with frozen weights, we find that pretraining gains are uneven: substantial for sleep staging and clinical EEG, but small or absent for motor imagery and event-related potentials, and almost entirely diminished once classifiers also receive handcrafted spectral and covariance features. To understand why, we pretrain encoders on transformed EEG that preserves only selected statistics of the original signals. Transformations retaining spectra and cross-spectra alone reproduce the benefits of original-EEG pretraining, yet the same transformation substantially impairs supervised motor-imagery and ERP decoding. Finally, adding a supervised term to reconstruction shows these representations to be learnable from the same data at negligible cost to the pretext objective; but removing supervision erodes most of the acquired decoding gains with little change in reconstruction loss. In the configurations studied, good reconstruction therefore does not assure the acquisition or retention of representations useful for motor-imagery and ERP decoding. These findings highlight objective–task alignment as a central challenge for EEG pretraining, motivating objectives better aligned with downstream tasks.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG, Foundation Models, Self-Supervised Learning, Representation Learning, Pretraining Objectives

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Mg2COpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

In this work, the authors compared different EEG FMs with their randomly initialized versions and tried to isolate the contribution of self-supervised pretraining across diverse downstream tasks. They find that the gain is task-dependent. Then, they investigate which signal properties support pretraining versus downstream decoding. Their results suggest that current reconstruction- and prediction-based pretraining objectives can rely heavily on spectral and cross-spectral information, while failing to reliably acquire or retain temporal/task-relevant representations needed for MI and ERP decoding.

Critical strengths
  1. It is well motivated to investigate and diagnose the pretraining’s contribution, and the authors have found interesting results regarding the mismatch between the pretraining task and downstream tasks.
  2. The experimental design is generally careful and well controlled for the specific hypotheses being tested.
Critical weaknesses
  1. I appreciate the efforts and the amount of work the authors have put into the benchmark. As stated in the abstract, the broad benchmark covering 23 encoders and 34 datasets can convincingly establish task-dependent pretraining gains. However, the paper’s main mechanistic conclusions are not derived from this large-scale benchmark, but from a much narrower set of controlled experiments centered primarily on REVE and a small subset of datasets. It therefore remains unclear whether the results and mechanistic observations generalize across diverse architectures, particularly models with substantially different tokenization schemes, pretraining objectives, and sizes. In this sense, the headline emphasis in the abstract on “23 encoders and 34 datasets” may overstate the breadth of evidence supporting the paper’s mechanistic claims, since the large-scale benchmark mainly establishes the descriptive phenomenon, while the follow-up diagnosis is validated in a substantially narrower setting.

  2. Given that another central motivation for foundation models is scaling, this study would benefit from explicitly analyzing whether the observed objective–task mismatch changes with model size, pretraining data, or compute. The current REVE-small/base and 5×-budget experiments partially answer this, but I find it hard to conclude that scaling cannot recover the missing MI/ERP information.

  3. The visualization and presentation of the results could be improved. Given the rich results from the benchmark and the large number of carefully controlled experiments with paired data, I believe the paper could extract and present considerably more insight from the collected results. For example, it would be informative to examine how pretraining gain varies with encoder scale, model capacity, pretraining data size, or compute, and whether these relationships differ across task families. Also, the contribution of different preserved signal properties could be summarized more systematically across downstream tasks rather than being presented primarily through a limited set of aggregate comparisons. Even the current main figures could be made more informative. In Figure 1, the pretrained-versus-random gain is useful for isolating the contribution of pretraining, but presenting an absolute supervised reference (e.g., EEGNet) alongside it would help distinguish between tasks where pretraining adds little because the task is already well captured by simple representations and more challenging tasks. In Figure 2, the paired nature of the comparisons could be better exploited by reporting effect sizes and confidence intervals, or formal significance/equivalence analyses, directly in the main figure.

  4. Longer reconstruction training does not necessarily guarantee that all task-relevant features have been learned. Reconstruction may first capture dominant, easier-to-model low-frequency structure, while weaker MI-relevant mu/beta dynamics or temporally localized patterns may emerge only later. Tracking frequency- and time-specific representation quality throughout pretraining would therefore help rule out the possibility that the current models are simply undertrained or underfitting the reconstruction objective. As a useful limiting thought experiment, if a masked-reconstruction model could perfectly reconstruct the full EEG signal from its visible context, then the encoder–decoder system would necessarily have captured essentially all predictable signal structure, including information relevant to MI and ERP. From this perspective, the weak downstream transfer observed here could reflect not only the objective–task misalignment, but also the fact that current models are still far from this reconstruction limit. The manuscript would benefit from distinguishing these two possibilities.

  5. The paper diagnoses the problem well but provides limited guidance on how it might be addressed. Although developing a complete solution may be outside the scope of the study, I would expect a more detailed discussion of possible self-supervised objectives that could better preserve temporal or task-relevant information for MI and ERP. In particular, the fact that the same failure persists under the tested JEPA objective suggests that the issue may extend beyond masked reconstruction, and the manuscript could discuss more explicitly what this implies for the design of future EEG pretraining objectives.

Questions for authors

Regarding Critical Weaknesses (CW) 1:

The main mechanistic analyses are conducted in a much narrower setting, primarily using REVE and six downstream datasets. This raises several important questions. To what extent do the signal-transformation findings generalize beyond REVE? Do models with substantially different tokenization strategies or self-supervised objectives (for example, contrastive approaches rather than MAE- or JEPA-style prediction) show the same dependence on spectral and cross-spectral information? Similarly, does the acquisition/retention behavior observed in Figure 3 also hold for other EEG foundation-model architectures? These questions are important because the paper’s broad benchmark establishes that pretraining gains are task-dependent, but the proposed explanation for why those gains differ is validated much more narrowly. I also suggest having the abstract more clearly distinguish the evidence from the broad benchmark from the evidence supporting the mechanistic conclusions.

Regarding CW2:

  1. Have the authors compared generic/random 4-s pretraining windows with event- or cue-aligned windows?

The current experiments use a common 4-s windowing scheme. However, for ERP and, to some extent, MI, much of the discriminative structure is defined relative to an event or cue. If these recordings are segmented without preserving a consistent temporal reference, task-relevant dynamics may be diluted across windows, especially when most segments in the pretraining data pool are not time-locked ERPs. It would therefore be informative to compare SSL pretraining on generic 4-s segments with pretraining on stimulus-/cue-aligned windows while keeping the architecture, objective, and amount of data fixed.

  1. Following question 1, this study shows that reconstruction on the same downstream corpus does not recover the supervised gain, but this does not fully answer whether the type of data and the structure of the unlabeled data matter. One important distinction is between using “more MI/ERP recordings” and using many unlabeled epochs that repeatedly preserve the same event-relative features. Therefore, how confidently can the weak MI/ERP transfer be attributed to objective–task mismatch rather than insufficient exposure to temporally aligned task structure? The authors interpret the Section 4 results as evidence that the objective is a key bottleneck because the same architecture and recordings can learn strong MI/ERP representations once supervision is added. Could a large collection of task-structured but still unlabeled MI/ERP epochs allow the same SSL objective to learn temporal information that is currently missing?

Given the scope of the comments above, I recognize that there are quite a few questions and do not expect the authors to address all of them. I would prioritize improving the result presentation and visualization (CW3) and making fuller use of the existing broad benchmark through additional analyses (CW1). Further experiments addressing the generalizability concern in CW1 would be a valuable addition, but even a small number of validations across different architectures or objectives would considerably strengthen the robustness and broader relevance of the mechanistic conclusions.

7 comments
AuthorsOpenReview ↗

Response to Reviewer Mg2C (1/4): CW1

Thank you for the detailed and constructive feedback.

CW1 — Beyond REVE

The mechanistic results rest mostly on REVE and six datasets. Do they hold for other architectures, tokenisations and objectives (e.g. contrastive)? Does acquisition / retention (Fig. 3) hold elsewhere? The abstract should separate the benchmark evidence from the mechanistic evidence.

To address this point, we extended both analyses well beyond REVE (new App. K, new Fig. 16, p. 69). Specifically, we included all 23 published encoders of our benchmark and all 27 datasets of the five main families, instead of six. We also pretrained five new encoder–objective pairs: CBraMod (criss-cross attention over per-electrode patches) and LUNA (electrodes compressed into learned queries) with masked reconstruction, REVE-small with a contrastive objective (SimCLR), and BIOT and BENDR with their own contrastive objectives. Finally, we repeated acquisition / retention (Fig. 3, Section 4, p. 8) with CBraMod and LUNA.

The new experiments support the findings of Section 3 across architectures, tokenizations and objectives, contrastive ones included. Acquisition / retention (Section 4) holds for two other architectures.

Section 3 (pretraining on surrogates).

  1. Published weights (Fig. 16a). As suggested, we evaluated the 23 published encoders on surrogate versions of the 27 datasets that keep only their spectra and cross-spectra. If pretraining had learned more than these statistics, its gain (originally published weights minus a random copy) would shrink on the surrogates. This is not the case: in every family of tasks, the average gain changes by at most 0.007 κ, and across encoders, the gains on EEG and on surrogates correlate at r = 0.95. The test can fail: with surrogates that keep the spectra but not the cross-spectra, the gain drops on clinical EEG (by 0.036 κ) and on SSVEP (by 0.083 κ).
  2. New pretrainings (Fig. 16b). Pretraining all 23 encoders twice, on EEG and on surrogates, was not possible during the rebuttal period, so we did experiments for the five new pairs only. We read each one frozen on the six datasets of Section 3 (three seeds; one for BENDR; BIOT on the four datasets that fit its bipolar montage), comparing against REVE-small (masked reconstruction) re-run as the reference. Note that, if spectra and cross-spectra are enough for transfer, the last column should be close to zero:
Encoder, objectiveGain from pretraining on EEGGain from pretraining on surrogatesDifference (surrogates − EEG)
REVE-small, masked reconstruction (re-run)+0.22+0.21−0.008 ± 0.006
CBraMod, masked reconstruction+0.15+0.11−0.035 ± 0.059
LUNA, masked reconstruction+0.07+0.08+0.001 ± 0.007
REVE-small, SimCLR+0.10+0.10−0.003 ± 0.006
BIOT, contrastive−0.20−0.23−0.029 ± 0.022
BENDR, contrastive−0.003+0.03+0.036

For the four encoders that gain from pretraining, the difference is close to zero. CBraMod's larger difference comes from one SSVEP corpus, where its attention readout is unstable across seeds. Without this corpus, the difference is −0.002. On this corpus, a readout that keeps the electrodes separate gives κ = 0.79 for both pretrainings. On all 27 datasets, original EEG keeps a small edge for these four encoders (0.007 to 0.015 κ), as in the paper's REVE runs (0.009). BIOT and BENDR do not gain from pretraining, like their published weights, so there is no gain to explain.

Section 4 (acquisition and retention, Fig. 16c–d). We repeated Fig. 3 with CBraMod and LUNA on MI, and with CBraMod also on ERP. The pattern is the same as with REVE. A small label term added to reconstruction raises frozen decoding to 0.36–0.43 κ. Reconstruction alone then erases most of this gain (at most 14% is kept), while continued supervision keeps it.

All these results are in the new Fig. 16 (App. K, p. 69 of the revised PDF): published encoders read on surrogates (a), new pretrainings (b), acquisition and retention with CBraMod and LUNA (c, d).

AuthorsOpenReview ↗

Response to Reviewer Mg2C (2/4): CW3

CW3 — Presentation

Extract more from the existing results (gain against encoder scale, data and compute, by task family). Fig. 1: add an absolute supervised reference such as EEGNet. Fig. 2: show effect sizes and intervals, or significance / equivalence tests.

We updated both main figures and added the analyses you list.

Fig. 1 now shows absolute scores, with EEGNet as the supervised reference. Mean κ over the published encoders and datasets of each family:

Task familyRandom encoderPretrained encoderEEGNetDistance to EEGNet covered
Sleep0.270.430.5363%
Clinical0.260.360.4363%
SSVEP0.120.180.4616%
Motor imagery (MI)0.070.100.419%
ERP0.020.020.26none

MI and ERP are far from solved, so the small gain there is not a ceiling effect.

Revised Fig. 1 (Section 2.1, p. 4): absolute scores with EEGNet (a), gain from pretraining (b), gain once descriptors are given to both heads (c).

Fig. 2 now shows paired differences with 95% intervals and an equivalence test. Pretraining on PSD + cross-spectra surrogates (phase-randomized windows that keep their spectra and cross-spectra) instead of original EEG changes transfer by +0.002 κ [−0.008, +0.012]. This is equivalent to zero within ±0.02 κ, 10% of the pretraining gain (two one-sided tests, p < 0.001), and so are the other four objective, size and budget configurations. The surrogate does remove task information: it costs a supervised EEGNet 0.30 κ on MI and 0.34 on ERP.

Revised Fig. 2 (Section 3.2, p. 7): scores by signal version (a) and paired differences with 95% intervals and the ±0.02 equivalence band (b).

Also new in the appendix:

  • The gain against parameters, pretraining hours and compute of the published encoders, by family (new Fig. 8, App. E.5, p. 45). No association survives correction, but with 10 to 22 encoders per panel this excludes only strong trends; the controlled runs of CW2 answer more directly (new Fig. 7, App. E.5, p. 44);
  • By task, a table with the share of the pretraining gain and of EEGNet's score that each signal property keeps (App. E.4, Table 29);
  • Formal statistical significance tests of the task dependence (App. B.1; 69Vo, W3): sleep and clinical gains exceed MI and ERP gains by +0.12 κ [0.07, 0.19], permutation p < 0.001.
AuthorsOpenReview ↗

Response to Reviewer Mg2C (3/4): CW2 and its two questions

CW2 — Scaling (App. E.5, new Fig. 7, p. 44)

Does the objective–task mismatch change with model size, pretraining data or compute?

Within the scales we could test, none closes the MI and ERP gap. We pretrained on four amounts of distinct TUEG data: 22 h, 222 h, 2,222 h and the full 23,000 h. All four runs see the same number of windows (42 M, about two epochs of the full corpus), so smaller sets are repeated more: each window of the 22 h set is seen about 2,000 times. More data stops helping after 222 h. Longer training adds little: we followed one run for 43 epochs of the full corpus (about 900 M windows), and it barely changes after the first epoch. More compute or parameters helps MI a little and ERP hardly at all, while sleep and clinical EEG already match EEGNet (six datasets of Section 3):

Task familyREVE-small (paper)REVE-small, trained 5× longerREVE-base, 5.4× more parametersEEGNet (supervised)
Motor imagery0.120.130.130.42
ERP−0.010.000.000.34
SSVEP0.720.770.750.89
Sleep0.540.560.550.54
Clinical0.590.630.630.53

Larger scales (more than 69 M parameters, 23,000 h of data or 43 training epochs) remain untested; the conclusion now says so.

CW2-Q1 and Q2 — Cue-aligned windows, and unlabeled task epochs (App. F.7)

Q1: pretraining on generic 4-s windows vs cue- or stimulus-aligned windows, all else fixed. Q2: could many unlabeled, event-aligned MI/ERP epochs let the same objective learn what is missing?

We find window placement to make no difference, and having many more unlabeled aligned epochs barely helps. The experiments in Section 4 already consider pretraining on cue-aligned epochs: one 4-s epoch per trial, at a fixed position relative to the cue (the revised text now says so). These are about 24,000 unlabeled epochs of the target dataset, each seen about 500 times. We added the generic counterpart, with the same people, number of windows, model, objective and budget (three seeds; paired intervals contain zero):

Pretraining windows (same corpus)MI κERP κ
Generic 4-s windows0.070.00
Cue-aligned epochs0.060.00
Cue-aligned epochs, with labels0.370.28

With 7.5 times more ERP epochs, held-out reconstruction improves but ERP decoding stays at chance; with 2.1 times more MI epochs, κ rises only from 0.065 to 0.088. The information is in the aligned epochs, and reconstruction does not extract it. A contrastive objective on the same MI epochs does better (κ = 0.22), and nearly as well (0.19) when trained on their PSD + cross-spectra versions: it too learns mostly spectra and cross-spectra (App. K).

Reviewer_wURmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper systematically investigates the contribution of pretraining to EEG foundation models and examines whether its benefits vary across downstream tasks. Through evaluations of 23 published encoder configurations across 34 datasets, the authors find that pretraining provides substantial improvements for sleep staging and clinical classification but limited benefits for motor imagery (MI) and event-related potential (ERP) decoding. Moreover, these improvements largely disappear when classical spectral and covariance features are incorporated. To further investigate this phenomenon, the authors conduct controlled experiments using transformed EEG signals and show that preserving spectral and cross-spectral statistics is sufficient to reproduce the benefits of pretraining on original EEG, despite the loss of information important for MI and ERP decoding. Additional experiments demonstrate that task-relevant representations can be learned through supervision with minimal impact on reconstruction performance, but these benefits substantially diminish when supervision is removed. Overall, the findings highlight the mismatch between pretraining objectives and downstream task requirements, suggesting the need for better-aligned self-supervised learning objectives.

Critical strengths

1.The paper provides a comprehensive and well-controlled evaluation of EEG foundation models, covering 23 encoder configurations and 34 datasets across diverse downstream tasks. By comparing pretrained encoders with randomly initialized counterparts under matched evaluation settings, the authors effectively isolate the contribution of pretraining from architectural factors. Furthermore, incorporating classical EEG features allows them to assess whether pretrained representations provide additional predictive information beyond established signal descriptors. These experiments offer valuable insights into the task-dependent benefits of EEG pretraining and strengthen the reliability of the findings. 2.The paper provides an insightful investigation into the mismatch between pretraining objectives and downstream task requirements through carefully designed experiments. By applying controlled signal transformations, the authors demonstrate that preserving spectral and cross-spectral information can maintain pretraining performance while substantially degrading supervised MI and ERP decoding. Furthermore, the experiments involving the removal of supervision show that reconstruction objectives alone fail to retain task-relevant representations acquired through supervised learning. These complementary experiments provide convincing evidence that successful reconstruction does not necessarily translate into effective downstream representations, offering meaningful insights for the development of future EEG pretraining objectives.

Critical weaknesses

1.The paper primarily evaluates pretrained representations using frozen encoders, which is appropriate for isolating the contribution of pretraining. However, a key motivation for EEG foundation models is their ability to facilitate adaptation across diverse downstream tasks. Although the authors provide limited fine-tuning and low-label experiments, they do not systematically investigate whether pretraining improves fine-tuning efficiency, convergence speed, or computational costs compared with training from scratch. Therefore, the observed limited gains in frozen representations may not fully reflect the practical value of pretraining, particularly in terms of efficient cross-task adaptation. 2.The paper demonstrates that the evaluated self-supervised objectives, particularly reconstruction-based objectives, fail to consistently capture and retain task-relevant information for MI and ERP decoding. However, these limitations may reflect a mismatch between specific pretraining objectives and downstream task requirements rather than a fundamental limitation of self-supervised EEG learning. Although several objectives are considered, the key intervention experiments are conducted within a limited range of architectures and training strategies. Consequently, it remains unclear whether different downstream tasks require task-specific self-supervised objectives or whether more comprehensive pretraining objectives could learn general-purpose representations that benefit diverse EEG tasks. This distinction is important for understanding the generalizability of the findings and their implications for designing EEG foundation models.

Questions for authors

1.The paper primarily evaluates pretrained representations using frozen encoders. Although limited fine-tuning and low-label experiments are provided, could the authors further clarify whether pretraining offers advantages over training from scratch in terms of convergence speed, fine-tuning efficiency, and computational cost across different downstream tasks? In particular, do the limited gains observed with frozen encoders necessarily translate into limited practical benefits when pretrained models are fully fine-tuned? 2.The paper identifies a mismatch between the evaluated pretraining objectives and downstream MI and ERP tasks. However, it remains unclear whether these limitations arise from the specific self-supervised objectives or reflect a broader challenge in learning general-purpose EEG representations. Given that the key intervention experiments focus on a limited range of architectures and objectives, do the authors believe that different downstream tasks require task-specific self-supervised objectives, or that more comprehensive pretraining objectives could learn representations transferable across diverse EEG tasks? What evidence from the current experiments supports either interpretation?

2 comments

Thank you for the review and for the two precise questions.

W1 / Q1 — Does pretraining help fine-tuning? (App. L, pp. 70–72)

Beyond frozen encoders: does pretraining speed up convergence, make fine-tuning more efficient or cut compute compared with training from scratch? Do small frozen gains mean small practical value?

Pretraining speeds up fine-tuning where frozen features already help, costs nothing extra, and does not turn the small MI and ERP gains into a practical advantage. We fine-tuned end to end REVE-base and the same architecture but with random initialization, with an identical search, on five task families and two label budgets (2,642 runs), and now report convergence and cost from these runs.

  • Convergence is faster, but only where frozen features already help. On SSVEP, sleep and clinical EEG, the pretrained model reaches 90% of its best validation score in 1–2.3 epochs, against 5–9 from random initialization. On MI, both reach most of their level within three epochs. On ERP, the pretrained model improves over 10–20 epochs, while the random copy stays near 0.1 κ.
  • Cost. An epoch costs the same for both initializations but early stopping ends the pretrained runs sooner, so a full-label run takes 83 s against 107 s from random initialization, on one A40 GPU.
  • Practical value on MI and ERP stays small. With all labels, fine-tuning raises the advantage of pretrained weights on MI, mostly because the random 69 M-parameter copy trains poorly on only 2,700 windows. The fine-tuned model still does not beat EEGNet trained from scratch, a network of about 3,000 parameters (MI 0.38 against 0.40, ERP 0.21 against 0.34). With 32 labels per class, its advantage on MI and ERP is near zero under every protocol (at most 0.013 κ). LoRA and LP-FT agree (69Vo Q1).

W2 / Q2 — The tested objectives, or SSL for EEG in general?

Is the conclusion specific to the tested objectives, or to self-supervised learning for EEG in general? Do different tasks need task-specific objectives, or could a more complete objective serve them all?

Indeed, our claim concerns the objectives we tested, and the revised conclusion now says so explicitly. Our evidence suggests that a more complete objective is possible, but it does not show whether it will be one general objective or several task-specific ones.

  • Against "one objective's flaw": the same pattern holds for every family we tested (masked reconstruction, latent prediction, discrete-token prediction, contrastive), across architectures, sizes, budgets and pretraining datasets. Each draws its benefit from spectra and cross-spectra. Even a contrastive objective trained on the MI corpus itself learns mostly second-order information: trained on PSD + cross-spectra versions of the same windows, it keeps most of its MI score (0.19 against 0.22).
  • For "a more complete objective is possible": with a small label term, the same encoder learns MI and ERP while its reconstruction stays within 1% of the reconstruction-only model (Section 4). Reconstruction therefore does not prevent the encoder from learning these tasks; it simply does not require it.
  • What remains open. Any candidate should reward structure that second-order statistics cannot explain, and specifically the part the task uses: an objective trained to tell windows from their surrogates did not decode MI better than reconstruction in a first test. The revised conclusion discusses these directions (see also Mg2C CW5).

I appreciate the authors’ additional experiments and efforts to address my concerns. While the new results partially alleviate my doubts, they do not fully resolve my concerns; therefore, I will maintain my original rating.

Reviewer_69VoOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The work investigates the core pretext learning of EEG foundation models and finds that sleep staging and clinical EEG gained from pretraining, while motor imagery and event-related potentials diminished. Results obtained from 23 encoders and 34 datasets were used to support evidence of the misalignment in EEG pretraining.

Critical strengths
  1. It's a meaningful step to explore the use of pretraining for EEG foundation models, instead of just pretraining and fine-tuning for specific tasks.
  2. Several pieces of evidence have been found through extensive experiments, including the diversity gain of different signal properties and the minor decoding gain of motor imagery and ERP from reconstruction pretraining.
Critical weaknesses
  1. Several settings are specific, such as the preprocessing, pretraining models, decoding models, or pretraining tasks, which may introduce bias into the settings.
  2. It would be beneficial to give a clear description of the pretraining and fine-tuning pipelines used for comparison, clarifying whether the model used a frozen encoder or fine-tuned part of the model.
  3. Statistic analysis is necessary to make sure the generalizability of the conclusive results.
Questions for authors
  1. Could you evaluate whether there is a difference between full fine-tuning and parameter-efficient adaptation in downstream tasks?
  2. Would the preprocessing affect the overall performance? We know some tasks, such as ERPs, are sensitive to preprocessing.
  3. Could you give some further discussion on the design of EEG foundation model pretraining based on current results?
  4. Would the pretraining data choice affect the overall performance? e.g., would that achieve different results for specific downstream tasks when choosing similar datasets in pretraining and diverse datasets in pretraining
2 comments
AuthorsOpenReview ↗

Response to Reviewer 69Vo (1/2): W1–W3

Thank you for the review. Each point led to a new analysis, detailed in the revised appendix.

W1 — Specific settings

Preprocessing, pretrained models, decoders and pretext tasks are specific choices that may bias the conclusions.

Thanks for pointing this out. We tested this effect directly. We varied each choice you mentioned, one at a time. Our findings indicate that none of them influence or change our conclusions.

  • Preprocessing. We add three pipelines that are closer to each encoder’s original preprocessing (Q2).
  • Pretrained models. We evaluate all 23 published encoders through surrogates. We also train five new encoder–objective pairs ourselves (App. K).
  • Decoders. We test three pooling schemes and four adaptation regimes: frozen, LoRA, LP-FT and full fine-tuning (Q1).
  • Pretext tasks. We cover the four main families: masked reconstruction, latent prediction (JEPA), discrete-token prediction (LaBraM) and contrastive learning.
  • Data and scale (beyond the reviewer’s list). We pretrain on three datasets, one of which includes the target recordings. Data size goes from 22 h to 23,000 h, a range of more than 1,000×, and training lasts up to 43 epochs (Q4).

Across all these variations, our conclusions remain. As such, our findings do not hinge on one particular choice.

W2 — Pipelines

Describe clearly what is compared: a frozen encoder, or a partly fine-tuned one?

Every experiment in the main part of the manuscript uses a frozen encoder. Only an attention query and a linear layer are trained on the target labels, identically for the pretrained and the random encoder. Encoder weights change only in the fine-tuning experiments of App. L. A new table (App. A.6, Table 5, cited in Section 2) lists, for each experiment, which weights are pretrained, which are trained on labels and on which subjects, and whether the encoder is frozen.

W3 — Statistics (App. B.1, Table 8)

Statistics are needed to support the generality of the conclusions.

We added formal statistical significance tests. Gains are larger on sleep and clinical EEG than on MI and ERP by +0.12 κ [0.07, 0.19]. This holds under a permutation of task labels across datasets (p < 0.001), within encoders (16 of 19, Wilcoxon p = 0.001), and after removing the encoder–dataset pairs that overlap with pretraining. After Holm correction, the family gains are statistically significant on sleep and clinical EEG, but not on SSVEP, MI or ERP. Once descriptors are given to both heads, the gain is equivalent to zero within ±0.02 κ in all five main families (TOST).

AuthorsOpenReview ↗

Response to Reviewer 69Vo (2/2): Q1–Q4

Q1 — Full fine-tuning vs parameter-efficient adaptation (App. L, Table 54 and revised Fig. 17, pp. 71–72)

Is there a difference between full fine-tuning and parameter-efficient adaptation on downstream tasks?

We compare four regimes on REVE-base and its random copy, on five task families and two label budgets. Both encoders get the same head and the same hyperparameter search.

Gain from pretraining (pretrained minus random copy, κ, mean over the five families):

AdaptationEncoder parameters trainedAll labels32 labels per class
Frozen (main text)0%+0.18+0.21
LoRA, rank 82% (adapters)+0.18+0.12
LP-FT (head first)100%+0.17+0.21
Full fine-tuning100%+0.17+0.13

With all labels, the four regimes give about the same advantage. With 32 labels per class, full fine-tuning and LoRA lose about 40% of it, mostly on SSVEP, while LP-FT keeps it. No regime takes the pretrained model above EEGNet on MI or ERP: after full fine-tuning, MI reaches 0.38 against 0.40 for EEGNet, and ERP 0.21 against 0.34.

Q2 — Preprocessing (App. A.7, Table 6)

Does preprocessing matter? ERPs, for example, are known to be sensitive to it.

We re-evaluated 20 published encoders and their random copies, on 7 MI and 4 ERP datasets, under three preprocessings closer to the encoders' own: 0.3–75 Hz with a notch, amplitude kept, and both. The pretraining gain moves by at most 0.01 κ, and every interval contains zero. The ERP information is there under every preprocessing: EEGNet reaches 0.23–0.26 κ on the same windows, while pretrained encoders stay at 0.00–0.01.

Q3 — Implications for design (Conclusion)

What do the results mean for designing EEG pretraining?

(1) Within the ranges we tested, scale and similar data do not fix MI and ERP. (2) The objective should reward structure beyond spectra and cross-spectra, and specifically the part the task uses: the most direct attempt, telling windows from their surrogates, failed in a first test. (3) Weak supervision works where self-supervision does not, and event markers recorded with the data could provide it. (4) Evaluate per task family, against a random copy and a supervised baseline. The revised conclusion develops these points.

Q4 — Choice of pretraining data (App. E.5, Table 31)

Does the choice of pretraining data matter, e.g. data similar to the downstream task against diverse data?

It affects the magnitude of the gains, not which tasks gain in performance. Same encoder, recipe and budget, on three datasets (gain over the untrained encoder, κ; one run each):

Pretraining corpusMIERPSSVEPSleepClinical
TUEG (clinical, 23,000 h)0.070.060.110.340.31
Stieger2021 (MI only, 615 h)0.030.010.120.330.24
23 public datasets, all families (769 h)0.060.030.130.360.27

The task dependence is the same for all three, and MI-only data gives the lowest MI gain. The mixture contains the unlabeled recordings of six downstream datasets, yet this barely changes their MI or ERP gain. Similar data helps where the objective already helps: on TUAB and TUEV, from the same archive as TUEG, TUEG gives 0.09 κ more than the mixture. Section 4 takes "similar" to its limit by pretraining on the target windows themselves (Mg2C CW2-Q2).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their careful and constructive reviews and for their many concrete suggestions. We are very encouraged to see that all three reviewers found our work well motivated, designed and executed.

Three themes come up in several reviews: whether the mechanistic findings extend beyond REVE (Mg2C CW1, 69Vo W1, wURm W2); whether the results can be shown against a supervised reference and with formal statistics (Mg2C CW3, 69Vo W3); and whether the conclusions hold under other choices of scale, data, adaptation and preprocessing (Mg2C CW2/CW4, 69Vo Q1/Q2/Q4, wURm W1).

We address each with new analyses and experiments, summarized below. The new results are consistent and support our paper's conclusions, while improving the scope and overall presentation of our paper. All new material is highlighted in blue in the revised manuscript, and the page numbers below refer to it:

  1. Beyond REVE (new App. K, new Fig. 16, p. 69). The paper showed that the pretraining gain comes from spectra and cross-spectra, on six datasets and mainly with REVE variants (LaBraM in the appendix). We now test this on 27 datasets, with the 23 published encoders of Section 2 and five new encoder–objective pairs, and it holds. First, as Mg2C suggested, we evaluated the 23 published encoders on surrogate versions of the 27 datasets that keep only their spectra and cross-spectra. Their gain over a random copy barely moves: in every task family, the average gain changes by at most 0.007 κ. Across encoders, the gains on EEG and on surrogates correlate at r = 0.95. Second, we pretrained CBraMod and LUNA with masked reconstruction, and REVE-small, BIOT and BENDR with contrastive objectives, each on EEG and on surrogates. BIOT and BENDR gain nothing from pretraining here, nor do their published weights. For the other three, as for REVE, pretraining on surrogates instead of EEG lowers the score by at most 0.015 κ, while pretraining itself raises it by up to 0.15 κ. Finally, we repeated the experiment of Fig. 3 (Section 4, p. 8) with CBraMod and LUNA. As with REVE, a small label term added to reconstruction teaches them motor imagery (CBraMod also ERP), and continuing with reconstruction alone then erases most of it (at most 14% kept).
  2. Presentation and statistics (revised Fig. 1, p. 4, and Fig. 2, p. 7; App. B.1, E.4). Fig. 1 now shows absolute scores with EEGNet as a supervised reference. Fig. 2 now shows paired differences with 95% intervals and an equivalence margin. The task dependence and the near-zero gain beyond descriptors are now tested formally.
  3. Scale and data (App. E.5 with new Figs. 7 and 8, pp. 44–45; App. F.7). We varied the amount of pretraining data (four sizes, from 22 h to 23,000 h), the training length (up to 43 training epochs), the model size (up to 69 M parameters) and the pretraining corpus (three datasets). None of these changes the task dependence, and cue-aligned pretraining windows do not help MI or ERP.
  4. Undertraining (new App. G, new Fig. 10, p. 59). In the configurations we could test, training longer or with a larger encoder barely changes MI and ERP decoding, and reconstruction quality does not decide it.
  5. Adaptation and preprocessing (App. L with revised Fig. 17, p. 72; App. A.7). We add LoRA next to frozen, LP-FT and full fine-tuning, report convergence and cost, and test three other preprocessings. The conclusions hold.
  6. Future objectives (Conclusion). A short paragraph on objective design: two directions, and how to evaluate candidate objectives.

Wording. The Introduction said that PSD + cross-spectra pretraining transfers "equally well on every task family". It now says "as well on average", and "within 0.007 κ in every task family" for REVE-small with masked reconstruction. The abstract now separates the benchmark evidence from the pretraining experiments (Mg2C CW1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.