- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The manuscript addresses an important and timely problem concerning the evaluation of pretrained EEG foundation models on small, out-of-distribution datasets. The authors propose an evaluation framework intended to distinguish genuine transfer learning from inflated classification performance resulting from methodological choices, recording artifacts, or demographic confounding. The framework is applied to eight pretrained EEG foundation models using a newly released ear-EEG dataset comprising 23 participants with and without sound-sensitivity disorders.
Critical strengths
I appreciate the authors' efforts to address common methodological pitfalls in EEG-based machine learning, particularly their emphasis on transparent evaluation procedures and the use of multiple control analyses. The release of the ear-EEG dataset and accompanying analysis pipeline is also a valuable contribution.
Critical weaknesses
I have several concerns regarding the methodological novelty, the quality and characterization of the experimental data, and the interpretation of the findings. In particular, the limited sample size, substantial demographic confounding, insufficient documentation of the auditory paradigms, and certain aspects of the control analyses make it difficult to draw strong conclusions about the transferability of pretrained EEG representations.
-
Although the manuscript addresses an important methodological problem, I find the novelty of the proposed evaluation framework somewhat limited. Most of its components, including nested cross-validation, permutation testing, demographic confound analyses, shortcut detection, and synthetic positive controls, are well-established methodological approaches. The main contribution therefore appears to be the integration of these existing methods into a unified evaluation protocol rather than the development of a fundamentally new methodology. While systematically combining these approaches is certainly valuable, it remains unclear what specific methodological advances the proposed framework introduces beyond existing evaluation practices.
-
The description of the auditory paradigms is insufficient to adequately evaluate the experimental design and interpret the resulting EEG signals. The authors report three auditory tasks: a Listening Effort Test (LET), a Hearing Loudness Test (HLT), and an Aversive Sound Test (AST). However, several important methodological details remain unclear. Furthermore, the manuscript repeatedly characterizes the experimental paradigm as passive auditory stimulation. However, listening-effort and loudness-assessment paradigms frequently involve active perceptual judgments or behavioral responses. The authors should therefore clarify the exact instructions given to participants and whether any behavioral responses were required. If participants were asked to evaluate stimulus loudness, intelligibility, or aversiveness, the characterization of the paradigm as entirely passive would require reconsideration. In addition, the three auditory tasks likely involve different perceptual and cognitive processes. Since the authors aggregate EEG features across these paradigms, it would be informative to report classification performance separately for each task to determine whether the observed effects are consistent across different forms of auditory stimulation.
-
I am concerned about the selection and interpretation of the relatively long EEG analysis windows. The available recordings extend from −5 to +7 seconds relative to stimulus onset, and the evaluated foundation models use input windows of four or five seconds. The rationale for employing such long windows in the context of auditory stimulus processing is not sufficiently explained. Conventional auditory evoked responses, including the P1–N1–P2 complex, primarily occur within the first few hundred milliseconds following stimulus onset. Although sustained auditory processing can extend beyond this period, depending on stimulus duration and task characteristics, the most prominent transient auditory evoked components are generally observed within the first second. Consequently, analyzing several seconds of EEG activity may dilute potentially informative stimulus-related responses with substantial amounts of activity that are not directly related to the initial auditory response. The authors should provide a stronger physiological justification for their window selection, particularly in relation to the duration and characteristics of the auditory stimuli. It would also be informative to investigate whether shorter, physiologically motivated post-stimulus windows yield more stable or meaningful classification results. This concern is particularly relevant given the reported near-chance performance when distinguishing pre-stimulus from post-stimulus EEG activity. It remains unclear whether these results reflect weak auditory responses in the recorded data, limitations of the pretrained models, or an analysis strategy that is insufficiently sensitive to transient auditory responses.
-
The authors initially recorded EEG data from 40 participants but excluded 17 participants due to insufficient signal quality, resulting in the final sample of only 23 participants. An exclusion rate of 42.5% is remarkably high and raises concerns regarding the overall quality and reliability of the EEG recordings. The high exclusion rate may indicate limitations of the recording setup, acquisition procedures, or quality-control criteria. Furthermore, excluding such a substantial proportion of the original sample may introduce selection bias, particularly if recording quality is systematically associated with participant characteristics.
-
The clinical and control groups differ substantially in both age and sex. The clinical group has a mean age of 41.7 years and comprises 11 males and five females, whereas the seven controls have a mean age of 25.3 years and are exclusively female. These pronounced differences make it difficult to determine whether classification performance reflects neurological characteristics associated with sound-sensitivity disorders or more general demographic differences. The authors acknowledge this limitation and provide additional analyses, which is appreciated. Nevertheless, the substantial group imbalance fundamentally limits the interpretation of the reported classification results. Given the small sample size, statistical adjustments or subgroup analyses cannot fully resolve this issue.
-
Furthermore, the clinical group combines individuals with tinnitus, hyperacusis, misophonia, and comorbid conditions. These are clinically distinct phenomena that may involve different neurophysiological mechanisms. The rationale for combining these conditions into a single classification category should therefore be explained more thoroughly. Importantly, the manuscript should clarify how the clinical diagnoses were established. Were the disorders formally diagnosed by qualified clinicians using standardized diagnostic procedures, or were participants classified based on self-reported symptoms or questionnaires?
-
One of the central conclusions of the manuscript is that BIOT-18ch's initially high classification performance (AUROC = 0.86) primarily originates from failed EEG channels. To support this conclusion, the authors demonstrate that classification performance decreases substantially when restricting the analysis to the three channels that pass quality control in all participants. While this represents an important control for channel-availability confounding, the interpretation of the result is not entirely straightforward. Reducing the input from 16 to only three channels simultaneously removes information about channel failures and eliminates a substantial proportion of potentially informative EEG data. Consequently, the observed decrease in classification performance cannot be attributed exclusively to the removal of recording-quality artifacts. The authors should consider additional analyses that disentangle these two explanations. For example, classification performance could be evaluated using matched subsets of channels or alternative approaches that preserve a larger proportion of the EEG signal while minimizing information about channel failures. Such controls would help determine whether the observed decrease in performance primarily reflects the removal of a spurious shortcut or simply the loss of relevant neural information.
-
Figure 3 demonstrates considerable fluctuations in classification performance when the analysis window is shifted by only 200 ms, despite substantial temporal overlap between adjacent windows. For LaBraM, classification performance varies from approximately 0.30 to 0.96 AUROC across different window positions. These fluctuations are striking and raise questions about the robustness of the extracted representations and the reliability of the classification results. Given that adjacent four-second windows contain largely overlapping EEG data, it is difficult to determine whether such substantial performance differences reflect meaningful temporal variation in neural information or statistical instability resulting from the extremely small sample size.
-
The manuscript contains considerable repetition, particularly regarding the central message that classification performance can be inflated by methodological choices, recording-quality artifacts, and demographic confounding. While this is an important point, it is reiterated several times with relatively little additional information. For example, the abstract, the "Protocol and findings" paragraph in the Introduction, and the subsequent "Contributions" paragraph all summarize essentially the same observations concerning BIOT-18ch, LaBraM, and the proposed evaluation protocol. Similarly, Section 5 presents the main findings regarding failed channels and window selection, which are subsequently described again in considerable detail in Lessons 1 and 2 of Section 6, and summarized once more in the Discussion and Conclusion.
Questions for authors
How does the actual experimental setup looks like? What exact auditory stimuli were presented in each task? What were their duration, intensity, and acoustic characteristics? Were stimulus intensities standardized across participants, or were they individually adjusted? How many trials were collected per condition, These details are particularly important when investigating individuals with sound-sensitivity disorders, as differences in stimulus intensity or subjective perception could substantially influence the recorded EEG responses.