acceptodds
Withdrawn from ICLR 2027

Which Data, Not How Much: Architecture-Dependent Data Selection for EEG Foundation Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Electroencephalography (EEG) foundation models have achieved strong performance across diverse tasks. Despite differences in architecture and training objectives, most share a common strategy: pretrain on increasingly large data pools to learn better general-purpose representations. This strategy is well established in natural language processing and computer vision, but whether more pretraining data consistently improves EEG transfer remains an open question. We examine 16 candidate datasets, 13 downstream tasks, and 202 configurations across LaBraM, CBraMod, and BIOT, keeping the backbone and pretraining epoch count fixed within each architecture. Adding sources can lower average performance: 121 configurations achieve higher mean test balanced accuracy than full-pool pretraining with fewer training windows. All 39 architecture–task combinations have a smaller, higher-scoring configuration; even adding sources from the target's task family or its own training partition need not help. Similarly sized mixtures yield different benchmark averages and opposing responses on related targets. Capacity experiments show recovery under further expansion at fixed Base capacity; increasing LaBraM to Large narrows the full-pool deficit but leaves a six-source subset ahead by 2.84 percentage points. We compare four practical rankings—Progressive, pairwise dataset valuation (PDV), Size, and Coverage—that identify useful mixtures at different scoring costs, with Size and Coverage sharing their rankings across LaBraM and CBraMod. The findings show that source composition matters alongside data volume and support choosing pretraining datasets using appropriate data-selection methods.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG foundation models, Pretraining data composition, Data selection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iGvMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether EEG foundation models consistently improve downstream transfer performance as the volume of heterogeneous pre-training data increases. The authors conducted controlled experiments using three representative EEG foundation models, LaBraM, CBraMod, and BIOT, across 16 candidate source datasets, 13 downstream tasks, and 202 source data combinations, while keeping the model backbone and number of pre-training epochs constant. The results demonstrate that pre-training on the full set of 16 datasets does not consistently yield optimal results; many smaller data combinations achieved higher balanced accuracy. Furthermore, identical data volumes with different data compositions could lead to significantly different performance outcomes.

Critical strengths
  1. This paper focuses on the data composition in the pretraining of EEG foundation models, which has certain research value. The authors conducted experiments on a variety of model architectures, candidate datasets, and downstream tasks. The research conclusions are not limited to a certain model or specific task, which makes the findings more convincing.
  2. The experimental analysis is comprehensive. It not only considers the data size, but also further analyzes the factors affecting the transfer performance by comparing different data combinations, model capacity and source domain sorting methods. In addition, the paper also evaluates the pool-external tasks, which further enhances the credibility of the source domain selection results.
Critical weaknesses
  1. The paper does not fully consider the randomness of the pretraining process itself. The three random seeds are only used for downstream fine-tuning of the same pretraining model. The authors did not conduct multiple independent pretraining runs on different data combinations. Therefore, the currently reported performance fluctuations and confidence intervals cannot reflect the randomness of the pretraining phase.
  2. The selection of some optimal data subsets in the paper depends on the results of the test set. According to Appendix B.3, due to the lack of verification records corresponding to different subset sizes, the author actually determined some of the optimal configurations based on the performance of the AVG13 test set. This approach may lead to overestimation of the reported performance improvement, and it is difficult to directly apply to actual scenarios where test results cannot be obtained in advance. The additional external task evaluation shown in the paper cannot completely solve this problem.
  3. The training budget between different data combinations has not been strictly controlled. In this paper, the number of pretraining rounds is uniformly set to 50 epochs. The combination of large data size will experience more training windows and optimization updates, so the actual training volume of different combinations is not the same. Although the author has carried out a certain degree of verification by comparing combinations with similar data sizes, for the key comparison between Full16 and smaller subsets, it is still necessary to supplement the experiment with the same number of optimization updates.
Questions for authors
  1. Can authors use multiple different random seeds for independent pretraining and report the variance of the pretraining phase for the full dataset and selected subsets? This helps to determine whether the performance differences observed between different data combinations are stable, rather than being affected by the randomness of pretraining.
  2. In practical applications, the results of the test set cannot be obtained in advance. How to determine the appropriate data subset size k? Can authors propose a selection method that only relies on validation sets or nested validation, and report the experimental results on the main benchmarks and external tasks under this method?
  3. The paper uses the same number of epochs for pretraining for all data combinations, which means that the larger data pool will experience more training windows and optimization updates. Can the author supplement the control experiments with the same number of optimization updates or comparable computational budgets for Full16 and some key subsets to further determine whether the performance difference comes from data composition, or different training volumes and optimization processes?
Reviewer_yCpMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how pretraining data scale and source composition affect EEG foundation model transfer performance. It evaluates 202 pretraining configurations across 16 candidate datasets, three architectures (LaBraM, CBraMod, and BIOT), and 13 downstream tasks. The study finds that smaller data subsets can outperform full-pool pretraining and compares four source-selection strategies including Progressive, PDV, Size, and Coverage. Additional experiments examine model capacity and transfer to six external tasks.

Critical strengths

S1. Important and practically relevant research problem.

The paper addresses pretraining data scale and composition in EEG foundation models, where substantial heterogeneity in recording configurations, populations, and task paradigms makes efficient data selection an important practical problem.

S2. Broad empirical coverage.

The evaluation spans 16 candidate datasets, three representative EEG foundation models, 202 configurations, and 13 downstream tasks. Comparisons of similarly sized mixtures, model capacity, statistical tests, and six external tasks provide multiple perspectives on the effects of data composition.

Critical weaknesses

W1. Limited methodological novelty and actionable insights.

The study focuses on dataset-level selection using mostly existing ranking ideas and heuristics. Despite the extensive experiments, it does not establish an explanatory theory or sufficiently generalizable principles for selecting pretraining sources. The title somewhat overstates the scope of the investigation.

W2. Insufficient discussion of relevant literature.

Section 2 overlooks important work on pretraining data mixture optimization and selection beyond EEG, including RegMix and Data Mixing Laws. Especially DoReMi is not referenced correctly, leaving raw ref commands. These studies are directly relevant to source interactions, mixture optimization, and performance prediction, making the novelty and positioning of this work insufficiently clear.

W3. Limited data characterization and selection scope.

The 16 pretraining sources are largely conventional EEG downstream benchmark datasets currently. The paper provides little analysis of their recording configurations, paradigms, or subject-level differences. Most experiments explore dataset combinations without systematically examining preprocessing, within-dataset sample quality, or sample-level selection.

W4. Confounded data composition and training budgets.

Changing the source pool also changes the implicit sampling proportions. TUAB dominates some mixtures, accounting for approximately 47% of Full16 windows and 80% of LaBraM Cov-H3 windows. Moreover, fixing 50 pretraining epochs does not control total signal duration, optimization steps, or computational cost. These factors complicate the separation of data scale and composition effects.

W5. Limited independent pretraining validation and scaling generalizability.

The reported 7,878 evaluations are downstream fine-tuning results, with three fine-tuning seeds per configuration rather than independent pretraining repetitions. The relatively limited and highly imbalanced source pool also restricts the generalizability of the scaling conclusions.

W6. Shared LaBraM tokenizer complicates subset comparisons.

Appendix B.2 states that LaBraM's tokenizer is trained on Full16 and subsequently frozen for all subset experiments. Therefore, excluded sources can still influence the tokenizer and pretraining targets. This complicates the interpretation of source selection and comparisons with CBraMod and BIOT, which do not use this tokenizer.

W7. Insufficient justification and anomalous behavior of Coverage.

The Coverage metric lacks a convincing theoretical justification and depends on unequal probe sizes. Appendix E.4 reports that SCD and SEED-V each activate only one tokenizer code. Together with their near-chance downstream performance, these anomalies raise concerns about encoding compatibility, metric validity, and evaluation pipeline reliability.

W8. Insufficient analysis of transfer and optimization mechanisms.

Although the paper observes negative transfer and non-monotonic performance changes, it provides little analysis of training dynamics, gradient conflicts, or cross-task interference. Consequently, the experiments offer limited explanations for why specific source combinations help or hurt, and few corresponding training recommendations.

Questions for authors

Q1 (W4) How are source sampling proportions controlled across mixtures? Can the authors separate composition effects from changes in training steps, signal duration, and computational budget?

Q2 (W6–W7) How does the Full16-trained tokenizer affect LaBraM subset comparisons? What explains the near-total codebook collapse for SCD and SEED-V, and has the corresponding preprocessing and evaluation pipeline been verified?

Q3 (W1, W8) Can the authors provide evidence connecting source interactions to training dynamics or gradient conflicts, and clarify what generalizable data-selection guidance follows from the experiments beyond the observed rankings?

Reviewer_PD3vOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors investigate how different compositions of pretraining datasets affect the downstream finetuning performance of EEG Foundation Models, to address the questions whether similar scaling as foundation models in other domains can be observed or whether there is a different recipe to compose an optimal source pretraining dataset. To do so, they fix a downstream evaluation protocol and model architecture and vary the number of pretraining datasets included and the way to choose them (best individual transfer, size,…). They conclude there is no dominant way to choose pretraining datasets, but that many of the configurations nevertheless favor a subset of source datasets as opposed to all of them.

Critical strengths

(1) A principled ablation study of which datasets to include in EEG foundation models, based on size or heterogeneity and observing which scaling laws do and do not hold in this domain is very relevant to the field if it is executed correctly, which I highly doubt in this case.

Critical weaknesses

(1) The most critical weakness is that the differences reported in this paper can simply not be separated from noise. For instance, Figure shows that adding a new source dataset of 140 windows to one composed of over 40,000 windows drops their main evaluation number by about 6%, which is larger than the gains they are reporting from using less dataset. I cannot believe that these 140 windows could have any significant impact on what the pretraining learned and thus conclude that this encodes nothing more than noise. Given this perspective (and other similar reports) I consider the other performance gaps in this paper noise as well. In general I believe that most conclusions drawn in this paper originate from noise, that the FMs are not even competitive to the reported results in the original paper, and a few datasets whose inclusion actively hurt performance.

(2) The paper at no point offers any attempts at explaining what they are observing. It is limited to reporting an enormous amount of tables and different combinations of configurations, but at no point consider if there would be a reason to reject some (probably broken) datasets at all, whether some datasets could transfer better than each other due to similar recording setups (such as the different SEEDS datasets included) and still offers no way to actually select an optimal composition of datasets to pretrain the FM, without evaluating test performance.

Questions for authors

(1) In Figure 4, can the authors explain why the addition of TUSL drops the AVG13 by almost 6%, despite this very small dataset contributing only 140 windows to a corpus of 40,000 windows? How could such a small dataset affect the downstream performance this much? Should this not be considered noise then?

(2) The Full16 FMs, which are considered the baseline here, are significantly weaker than their reported counterparts. For TUEV for instance, Labram and CBraMod report 64 and 66% balanced accuracy after finetuning, while this work reports 57% and even 50% for these respectively. Can the observed gaps not be an artifact of working with significantly less performing FMs?

(3) The AVG13 performance metric is based on the finetuning on the pretraining datasets themselves. Yet Appendix G also reports transfer to held-out cohorts. If these are available, why did the authors consider the performance on the pretraining datasets more relevant, while FMs are typically more concerned with generalization to new cohorts?

(4) BCIC-IV-2a, SCD and SEED-V appear to be broken entirely: only 12,1 and 1 codes are activated in the codebook. This would suggest that something has made them collapse here and would make it very likely that they can actively harm training. It can even be observed in Table 11 that simply dropping two of these datasets boosts AVG13 to 59%, which is competitive and similar to all other best configurations. Can the author repeat their experiments omitting these datasets and see if their conclusions still hold up?

(5) Can the authors include more explanations for what is happening in the paper? For instance, if similar channel layouts can help explain transfer, or whether other source properties predict when a dataset helps or hurts: number of channels, window length, recording paradigm, population (clinical vs. healthy), data volume,… .

(6) Since all setups use the exact same training procedure, regardless of amount of data, have all models been trained to convergence?

(7) The reported gaps here are found by directly evaluating the test performance post-hoc: they cannot be used to guide dataset source selection when pretraining a new FM. Can the authors comment on how exactly someone should proceed composing a dataset given there appears to be no dominant approach?

Comments on the paper

1 comment

We thank the reviewers and the area chair for their time and constructive feedback. After careful consideration, we have decided to withdraw this submission to allow further experimental validation and revision. We appreciate the helpful comments, which will guide our continued work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.