- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces a benchmark for EEG OOD detection, evaluate a broad range of methods, and investigate their value in two downstream clinical prediction tasks. In its framework, the paper uses SITN for OOD detection and calibrated MSP as a proxy for predictive uncertainty.
Critical strengths
-
The analysis is interesting. The comparison between mixed ID/OOD data and ID-only data in Figure 3 reveals an interesting difference: MSP can be informative about classification accuracy despite poorly separating ID from OOD, whereas SITN effectively detects the evaluated distribution shift but provides less information about accuracy within most ID samples. This offers a useful empirical perspective on the different roles of these scores.
-
Thorough analysis in TUAB dataset. The paper provides extensive supplementary analyses, including hyperparameter sensitivity, comparisons across classifier architectures, sample-size effects, and detailed downstream results across perturbation conditions.
Critical weaknesses
- Limited methodological novelty and an overstated title. The study primarily evaluates existing methods, and the proposed method has limited methodological innovation. The title’s use of “disentangling” appears stronger than the experimental evidence supports. The main experiments show that SITN and MSP differ in their ability to detect perturbed inputs and predict classification errors. However, model uncertainty is primarily operationalized through MSP-based confidence, without a clear definition of the notion of uncertainty being assessed. The results support a distinction between OOD detection and confidence-based error prediction, but the broader claim of disentangling OOD detection and model uncertainty estimation requires clearer conceptual definitions and more targeted empirical validation.
- Insufficient experimental breadth for a benchmark contribution. The coverage of datasets remains limited relative to the paper’s broad claims. The evaluation does not include representative EEG foundation models, such as LaBraM and CBraMod, leaving it unclear whether the main conclusions extend to representations learned through large-scale pretraining. Furthermore, OOD detection performance is assessed primarily using AUROC. Reporting complementary metrics, such as AUPRC, balanced accuracy would help. Finally, results across multiple independent training seeds are not reported, making it difficult to assess the stability of the findings.
Questions for authors
Could the authors clarify the justification for treating the proposed perturbations, particularly at low severity, as OOD conditions? Some perturbations may produce variations already present in ID EEG, so applying a transformation does not automatically establish a meaningful distribution shift.
Thank you for your thorough review. We appreciate that you found our analysis interesting and useful.
Notion of uncertainty
While we would argue that the results and their discussion clearly outline the notion of uncertainty that we observe to be reflected in scores from discriminative methods like the MSP, we agree that “model uncertainty” may not be the right terminology to use here.
We have updated the terminology to “predictive uncertainty” in accordance with the terminology of Hüllermeier and Waegeman [1] along with the following description to clarify it:
Discriminative methods like MSP, on the other hand, are not suitable for OOD detection, but reflect predictive uncertainty. Even with ample exposure to similar samples during training, certain cases can be ambiguous and difficult for the model to classify. MSP scores can capture such cases and provide valuable information about how well the model is likely to perform across different in-distribution samples.
With this clarification in terminiology, we believe our experimental evidence appropriately supports the paper's claims.
[1] Hüllermeier, E., Waegeman, W. Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Mach Learn 110, 457–506 (2021). https://doi.org/10.1007/s10994-021-05946-3
Further evaluations
We have added extended results tables with additional metrics (AUPR and FPR@95TPR) in Appendices A and B.
Regarding training seed variability, methods are compared using the same underlying models. This is a different situation than comparing separately trained models where training variability can obscure performance comparisons. Moreover, results are consistent across the separate training runs for the different datasets as well as multiple training runs on different data subsets (Appendix E).
Unfortunately, foundation models like LaBraM and REVE were trained with substantially different data preprocessing setups than what was used in our experiments (different sampling rate, different filtering). Clean comparisons to these models would therefore either require re-training of the foundation models with matched preprocessing or considerable adjustments to the experimental setup (re-definition of perturbation severities for resampling and filtering perturbations; re-evaluation of the current models with matched preprocessing). While we agree that evaluations with foundation models could be interesting, they are not required to support the claims of our work and could be better explored in future work.
Justification for perturbations at low severity levels
We agree that in principle certain small perturbations could simulate some natural variation. However, this is unlikely to be the case for the specific perturbations and lowest severity levels chosen in our experiments. For instance, even a single channel swap would not be consistent with natural variation. Importantly, our results are robust in that even if the lowest severity level was ignored, the conclusions are unchanged.