- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether EEG foundation models consistently improve downstream transfer performance as the volume of heterogeneous pre-training data increases. The authors conducted controlled experiments using three representative EEG foundation models, LaBraM, CBraMod, and BIOT, across 16 candidate source datasets, 13 downstream tasks, and 202 source data combinations, while keeping the model backbone and number of pre-training epochs constant. The results demonstrate that pre-training on the full set of 16 datasets does not consistently yield optimal results; many smaller data combinations achieved higher balanced accuracy. Furthermore, identical data volumes with different data compositions could lead to significantly different performance outcomes.
Critical strengths
- This paper focuses on the data composition in the pretraining of EEG foundation models, which has certain research value. The authors conducted experiments on a variety of model architectures, candidate datasets, and downstream tasks. The research conclusions are not limited to a certain model or specific task, which makes the findings more convincing.
- The experimental analysis is comprehensive. It not only considers the data size, but also further analyzes the factors affecting the transfer performance by comparing different data combinations, model capacity and source domain sorting methods. In addition, the paper also evaluates the pool-external tasks, which further enhances the credibility of the source domain selection results.
Critical weaknesses
- The paper does not fully consider the randomness of the pretraining process itself. The three random seeds are only used for downstream fine-tuning of the same pretraining model. The authors did not conduct multiple independent pretraining runs on different data combinations. Therefore, the currently reported performance fluctuations and confidence intervals cannot reflect the randomness of the pretraining phase.
- The selection of some optimal data subsets in the paper depends on the results of the test set. According to Appendix B.3, due to the lack of verification records corresponding to different subset sizes, the author actually determined some of the optimal configurations based on the performance of the AVG13 test set. This approach may lead to overestimation of the reported performance improvement, and it is difficult to directly apply to actual scenarios where test results cannot be obtained in advance. The additional external task evaluation shown in the paper cannot completely solve this problem.
- The training budget between different data combinations has not been strictly controlled. In this paper, the number of pretraining rounds is uniformly set to 50 epochs. The combination of large data size will experience more training windows and optimization updates, so the actual training volume of different combinations is not the same. Although the author has carried out a certain degree of verification by comparing combinations with similar data sizes, for the key comparison between Full16 and smaller subsets, it is still necessary to supplement the experiment with the same number of optimization updates.
Questions for authors
- Can authors use multiple different random seeds for independent pretraining and report the variance of the pretraining phase for the full dataset and selected subsets? This helps to determine whether the performance differences observed between different data combinations are stable, rather than being affected by the randomness of pretraining.
- In practical applications, the results of the test set cannot be obtained in advance. How to determine the appropriate data subset size k? Can authors propose a selection method that only relies on validation sets or nested validation, and report the experimental results on the main benchmarks and external tasks under this method?
- The paper uses the same number of epochs for pretraining for all data combinations, which means that the larger data pool will experience more training windows and optimization updates. Can the author supplement the control experiments with the same number of optimization updates or comparable computational budgets for Full16 and some key subsets to further determine whether the performance difference comes from data composition, or different training volumes and optimization processes?
We thank the reviewers and the area chair for their time and constructive feedback. After careful consideration, we have decided to withdraw this submission to allow further experimental validation and revision. We appreciate the helpful comments, which will guide our continued work.