acceptodds
Under review as a conference paper at ICLR 2027

Does It Really Transfer? Auditing Frozen EEG Foundation Models on Small Ear-EEG Data

Abstract

Pretrained foundation models are often evaluated by freezing them and training a small classifier on a new dataset. When the dataset is small and differs from the pretraining data, a high score can come from choices tuned on the test subjects or from cues unrelated to the task rather than from the representation. We propose a protocol that tells whether such a model really transfers. It records every setup choice, retests each result by repeating the search that produced it on permuted labels and by nested cross-validation, checks whether failed channels or age and sex alone predict the label, and injects known signals to confirm that real effects can be found. We apply it to eight EEG foundation models pretrained on scalp recordings and to Synapse, a wearable ear-EEG dataset of 23 subjects with a passive auditory paradigm and clinical labels, which we release. A standard evaluation finds two models that appear to transfer. The protocol shows that BIOT-18ch's score (0.86 AUROC) comes from the channels that failed quality control and falls to 0.46 without them, and that LaBraM's (0.92) does not rely on failed channels but cannot be confirmed: it holds only in a narrow range of input windows and cannot be separated from sex and age at this sample size. On the same recordings, the protocol confirms a transfer of known size injected into a random group of subjects and confirms nothing without it. We release the protocol, code, data and all results.

Primary area:
datasets and benchmarks
Keywords:
foundation models, transfer evaluation, frozen representations, selection bias, shortcut learning, EEG, ear-EEG, dataset

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_RLV1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript addresses an important and timely problem concerning the evaluation of pretrained EEG foundation models on small, out-of-distribution datasets. The authors propose an evaluation framework intended to distinguish genuine transfer learning from inflated classification performance resulting from methodological choices, recording artifacts, or demographic confounding. The framework is applied to eight pretrained EEG foundation models using a newly released ear-EEG dataset comprising 23 participants with and without sound-sensitivity disorders.

Critical strengths

I appreciate the authors' efforts to address common methodological pitfalls in EEG-based machine learning, particularly their emphasis on transparent evaluation procedures and the use of multiple control analyses. The release of the ear-EEG dataset and accompanying analysis pipeline is also a valuable contribution.

Critical weaknesses

I have several concerns regarding the methodological novelty, the quality and characterization of the experimental data, and the interpretation of the findings. In particular, the limited sample size, substantial demographic confounding, insufficient documentation of the auditory paradigms, and certain aspects of the control analyses make it difficult to draw strong conclusions about the transferability of pretrained EEG representations.

  1. Although the manuscript addresses an important methodological problem, I find the novelty of the proposed evaluation framework somewhat limited. Most of its components, including nested cross-validation, permutation testing, demographic confound analyses, shortcut detection, and synthetic positive controls, are well-established methodological approaches. The main contribution therefore appears to be the integration of these existing methods into a unified evaluation protocol rather than the development of a fundamentally new methodology. While systematically combining these approaches is certainly valuable, it remains unclear what specific methodological advances the proposed framework introduces beyond existing evaluation practices.

  2. The description of the auditory paradigms is insufficient to adequately evaluate the experimental design and interpret the resulting EEG signals. The authors report three auditory tasks: a Listening Effort Test (LET), a Hearing Loudness Test (HLT), and an Aversive Sound Test (AST). However, several important methodological details remain unclear. Furthermore, the manuscript repeatedly characterizes the experimental paradigm as passive auditory stimulation. However, listening-effort and loudness-assessment paradigms frequently involve active perceptual judgments or behavioral responses. The authors should therefore clarify the exact instructions given to participants and whether any behavioral responses were required. If participants were asked to evaluate stimulus loudness, intelligibility, or aversiveness, the characterization of the paradigm as entirely passive would require reconsideration. In addition, the three auditory tasks likely involve different perceptual and cognitive processes. Since the authors aggregate EEG features across these paradigms, it would be informative to report classification performance separately for each task to determine whether the observed effects are consistent across different forms of auditory stimulation.

  3. I am concerned about the selection and interpretation of the relatively long EEG analysis windows. The available recordings extend from −5 to +7 seconds relative to stimulus onset, and the evaluated foundation models use input windows of four or five seconds. The rationale for employing such long windows in the context of auditory stimulus processing is not sufficiently explained. Conventional auditory evoked responses, including the P1–N1–P2 complex, primarily occur within the first few hundred milliseconds following stimulus onset. Although sustained auditory processing can extend beyond this period, depending on stimulus duration and task characteristics, the most prominent transient auditory evoked components are generally observed within the first second. Consequently, analyzing several seconds of EEG activity may dilute potentially informative stimulus-related responses with substantial amounts of activity that are not directly related to the initial auditory response. The authors should provide a stronger physiological justification for their window selection, particularly in relation to the duration and characteristics of the auditory stimuli. It would also be informative to investigate whether shorter, physiologically motivated post-stimulus windows yield more stable or meaningful classification results. This concern is particularly relevant given the reported near-chance performance when distinguishing pre-stimulus from post-stimulus EEG activity. It remains unclear whether these results reflect weak auditory responses in the recorded data, limitations of the pretrained models, or an analysis strategy that is insufficiently sensitive to transient auditory responses.

  4. The authors initially recorded EEG data from 40 participants but excluded 17 participants due to insufficient signal quality, resulting in the final sample of only 23 participants. An exclusion rate of 42.5% is remarkably high and raises concerns regarding the overall quality and reliability of the EEG recordings. The high exclusion rate may indicate limitations of the recording setup, acquisition procedures, or quality-control criteria. Furthermore, excluding such a substantial proportion of the original sample may introduce selection bias, particularly if recording quality is systematically associated with participant characteristics.

  5. The clinical and control groups differ substantially in both age and sex. The clinical group has a mean age of 41.7 years and comprises 11 males and five females, whereas the seven controls have a mean age of 25.3 years and are exclusively female. These pronounced differences make it difficult to determine whether classification performance reflects neurological characteristics associated with sound-sensitivity disorders or more general demographic differences. The authors acknowledge this limitation and provide additional analyses, which is appreciated. Nevertheless, the substantial group imbalance fundamentally limits the interpretation of the reported classification results. Given the small sample size, statistical adjustments or subgroup analyses cannot fully resolve this issue.

  6. Furthermore, the clinical group combines individuals with tinnitus, hyperacusis, misophonia, and comorbid conditions. These are clinically distinct phenomena that may involve different neurophysiological mechanisms. The rationale for combining these conditions into a single classification category should therefore be explained more thoroughly. Importantly, the manuscript should clarify how the clinical diagnoses were established. Were the disorders formally diagnosed by qualified clinicians using standardized diagnostic procedures, or were participants classified based on self-reported symptoms or questionnaires?

  7. One of the central conclusions of the manuscript is that BIOT-18ch's initially high classification performance (AUROC = 0.86) primarily originates from failed EEG channels. To support this conclusion, the authors demonstrate that classification performance decreases substantially when restricting the analysis to the three channels that pass quality control in all participants. While this represents an important control for channel-availability confounding, the interpretation of the result is not entirely straightforward. Reducing the input from 16 to only three channels simultaneously removes information about channel failures and eliminates a substantial proportion of potentially informative EEG data. Consequently, the observed decrease in classification performance cannot be attributed exclusively to the removal of recording-quality artifacts. The authors should consider additional analyses that disentangle these two explanations. For example, classification performance could be evaluated using matched subsets of channels or alternative approaches that preserve a larger proportion of the EEG signal while minimizing information about channel failures. Such controls would help determine whether the observed decrease in performance primarily reflects the removal of a spurious shortcut or simply the loss of relevant neural information.

  8. Figure 3 demonstrates considerable fluctuations in classification performance when the analysis window is shifted by only 200 ms, despite substantial temporal overlap between adjacent windows. For LaBraM, classification performance varies from approximately 0.30 to 0.96 AUROC across different window positions. These fluctuations are striking and raise questions about the robustness of the extracted representations and the reliability of the classification results. Given that adjacent four-second windows contain largely overlapping EEG data, it is difficult to determine whether such substantial performance differences reflect meaningful temporal variation in neural information or statistical instability resulting from the extremely small sample size.

  9. The manuscript contains considerable repetition, particularly regarding the central message that classification performance can be inflated by methodological choices, recording-quality artifacts, and demographic confounding. While this is an important point, it is reiterated several times with relatively little additional information. For example, the abstract, the "Protocol and findings" paragraph in the Introduction, and the subsequent "Contributions" paragraph all summarize essentially the same observations concerning BIOT-18ch, LaBraM, and the proposed evaluation protocol. Similarly, Section 5 presents the main findings regarding failed channels and window selection, which are subsequently described again in considerable detail in Lessons 1 and 2 of Section 6, and summarized once more in the Discussion and Conclusion.

Questions for authors

How does the actual experimental setup looks like? What exact auditory stimuli were presented in each task? What were their duration, intensity, and acoustic characteristics? Were stimulus intensities standardized across participants, or were they individually adjusted? How many trials were collected per condition, These details are particularly important when investigating individuals with sound-sensitivity disorders, as differences in stimulus intensity or subjective perception could substantially influence the recorded EEG responses.

Reviewer_hY3xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an evaluation protocol for deciding whether a frozen EEG foundation model genuinely transfers to a small, out-of-distribution dataset. The protocol runs and reports the full grid of adaptation choices (channel adaptation, sampling rate, embedding replacement, input window, failed-channel treatment, probe). It then tests the selected result with a max-statistic permutation test and nested LOSO cross-validation. It also checks shortcut baselines such as failed channels and demographics and injects a signal that should elicit response to investigate whether the model dan discern it. They release a cEEGrid dataset to accompany this analysis. In comparison to a standard best-cell evaluation the protocol rejects most FMs and is undetermined for LaBraM.

Critical strengths

(1) The paper reports how significance on a limited dataset can be p-hacked by showing the full configuration grid, raising further awareness on the care we should take before declaring a model generalizes to small, out of distribution datasets.

(2) The paper reveals a significant potential shortcut to be learned even when failed channels are interpolated, showing quality control is highly relevant to guard against such behavior.

Critical weaknesses

(1) The proposed audit appears unreasonably severe: Table 18 indicates that nested LOSO at 80% power on this sample size would require a true AUROC of 0.96, which is far too high for any reasonable task defined on EEG. Similarly, the positive example, the injected response, required an injected waveform with a magnitude of over one median trial SD on every single channel before pre- vs post-onset could be reasonably decoded, which appears far larger than what a true single-trial N1 would look like.

(2) Related to this, the protocol is validated on a single dataset that is very small, extremely prone to shortcuts and demographically decodable at an AUROC of 0.91. To make the insights and lessons more general, the paper would require at least a second test set, which is less demographically skewed and has more potential to positively pass the audit.

Questions for authors

(1) Nested selection picks [−3, +1] s in 17 of 23 folds (appendix J). To we this would indicate LaBraM barely uses post-onset windows to perform its separation, does this too not indicate that it is employing a shortcut instead of the true signal?

(2) The authors heavily make use of abbreviated codes (i.e. P4 indicating an MLP head) which makes the paper very difficult to read in some places. I would suggest they avoid this as much as possible or at least use intuitive abbreviations instead.

(3) The hyperparameter grid for the probes show the standard amount of weight decay on the linear probe. However, when training heads on such small datasets, significant effects of the probe regularization have been shown in literature, typically requiring a lot more regularization strength. Can the authors investigate whether their results remain the same as they are under strongly regularized linear probes?

(4) According to appendix N, neither the models nor the grand average appear to detect an auditory evoked response. I would believe that the auditory response is normally visible in a grand average, even on cEEGrid recordings, could the authors visualize this to ensure the timing is consistent across the trials?

(5) Related to the weaknesses mentioned above, could the authors apply their protocol to a more realistic, less demographically biased second test set, which has more chance to reasonably pass the audit to show the generality of their findings and the usability of their protocol?

(6) While I could imagine that shifting windows can significantly alter the AUROC of the downstream task, the degree to which this happens is very surprising to me: a median of 0.17 per step of 0.2 seconds. Can the authors further elaborate on how this could possibly happen and maybe analyze how these small temporal shifts are translated into a shift in the FMs latent space?

Reviewer_UNd2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an evaluation protocol to determine if a pre-trained model transfers to a downstream task on a small dataset. The protocol includes leave-one-subject-out cross-validation for hyperparameter tuning and specific checks for the influence of channel artefacts or potential demographic confounders. Different EEG foundation models are evaluated under this protocol on a new ear-EEG dataset (released together with this publication) with 23 participants.

Critical strengths
  • The new dataset could be a valuable resource for the community (although probably not as a benchmark for EEG foundation models; see below)
Critical weaknesses
  • Lack of novelty. The paper describes and addresses two ways in which evaluations can be flawed and result in unreliable results of model transfer performance: (1) hyperparameters are tuned on the test data, or (2) the test split is not independent of the training data (for instance when labels are defined on the participant level, but segments from the same participant are present in both the training and test splits). Both cases are well known and avoiding them is standard machine learning practice.
  • A separate test is designed to diagnose if a model does not transfer because it is unable to pick up on the predictive patterns in the data. To do this, a specific known signal is injected into the data and the evaluation tests if the model is able to predict it. However, testing if a model is able to pick up on one particular pattern reveals very little about whether the model is able to learn the patterns relevant to the downstream task.
  • The released ear-EEG data may have limited relevance as a testbed for EEG foundation models. As the authors acknowledge, the ear-EEG data is very different from the standard scalp EEG that EEG foundation models are developed for. Just like the goal of such EEG foundation models is not to work well for ECG data or stock market time series, they are not designed to work for ear-EEG. The dataset also has a small sample size of 23 participants with labels on the participant level.
Questions for authors
  • If I understand correctly, in line 407 the authors allege that LaBraM (and other works) tune hyperparameters on test data and then report the best configuration as the final performance? Can the authors justify why they arrive at this conclusion? The LaBraM paper reports a single hyperparameter configuration across all downstream tasks and their implementations on git partition data into disjoint train-val-test splits.
  • In line 142 the authors write that the released dataset fills the gap of there not being a public ear-EEG dataset that combines a passive auditory paradigm with clinical labels. Just that something does not exist yet is not a very convincing motivation. Can you explain why such a dataset is useful?
  • How is subject-level AUROC computed? How are segment predictions aggregated to the subject level?
Reviewer_PSCcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors ask whether a model trained on scalp EEG can also detect useful signals in EEG recorded around the ear. They test eight models on recordings from 23 people. Their protocol checks whether a model’s score could be explained by recording quality, the choice of time window, or differences between the patient and control groups.

Critical strengths

The paper asks a question that matters for clinical use. Before relying on an EEG model to identify a disorder, we need to know whether its predictions depend on brain activity or on how the recording was made. Table 2 shows why this matters. In this cohort, knowing which electrodes failed was enough to predict the clinical label with an AUROC of 0.76. I appreciate that the authors make this risk concrete.

Critical weaknesses
  1. Limited methodological novelty. The protocol combines established techniques, including label permutation, nested cross-validation, and channel controls. Adapting these tools to ear EEG is useful, but I see the main contribution as an engineering solution for this task. The paper does not clearly identify a new methodological idea.

  2. Limited evidence beyond one small cohort. All eight models are tested on recordings from the same 23 people. The signal injection experiment shows that the protocol can detect a known signal in those recordings. How reliably it can assess transfer in another cohort or on another EEG task remains unclear.

  3. Unclear advantage over existing interpretability methods. The paper compares several EEG models and includes useful controls. What I would like to see is a comparison with an existing method that also examines what drives a prediction. For example, would a method that identifies influential channels flag the same recording-quality issue for BIOT-18ch? Such a comparison would show what the proposed protocol adds.

Questions for authors
  1. Which existing approach comes closest to your check for reliance on failed channels? Using BIOT-18ch as an example, what could a reviewer learn from your full protocol that they would miss with that approach alone?
  2. Which conclusions have you validated beyond the 23 participants in SYNAPSE, and which do you consider specific to this case study?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.