acceptodds
Under review as a conference paper at ICLR 2027

What masking geometry works best for EEG foundation models?

Abstract

EEG foundation models hold promise for scalable brain-signal decoding across clinical and cognitive neuroscience applications, yet their pre-training pipelines remain poorly understood. Among design choices, the masking strategy is particularly critical: it determines what the network must predict and from which context. Yet it has never been ablated in isolation, as each new model bundles a new masking strategy with a new backbone and objective. In this paper, we formalize the design choices for spatio-temporal masking strategies and train various models with a single pipeline under varying masking configurations across two SSL frameworks (MAE and JEPA). We then systematically evaluate the resulting 58 pre-trained models on the 12 datasets of OpenEEGBench under a linear probe. Both frameworks agree on an optimal masking configuration and on shared failure modes. Outside these, performance is robust: 11 MAE and 9 JEPA configurations are statistically indistinguishable from the best. We further identify a novel JEPA-specific failure mode, tagged bias-inflation collapse, invisible to standard detectors. With a well-chosen mask, we reach REVE-level downstream performance at a fraction of its pre-training compute.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG, Foundation models, SSL, JEPA, MAE, Masking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_44bhOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates 29 masking geometries under MAE and JEPA using a shared EEG pre-training pipeline and frozen probes on 12 datasets. It identifies promising masking regimes and investigates a position-dominated JEPA failure mode termed bias-inflation collapse.

Critical strengths
  1. Systematic comparison. The shared pipeline, broad masking grid, and additional backbone and ratio experiments provide useful evidence beyond comparisons of unrelated models.
  2. Informative collapse analysis. Feature-mean/residual decomposition and band-power probing reveal position-dominated representations that the implemented training diagnostics miss.
Critical weaknesses
  1. Geometry and masking ratio are incompletely separated. Realized ratios span 50.8%–70.0% (Table 5). Appendix H.2 provides useful sensitivity checks, but random masking reaches 0.83 under MAE versus 0.82 for the recommended geometry after ratio tuning. The blanket recommendation against random masking is therefore unsupported.
  2. Claims exceed the measured uncertainty and interventions. Each configuration has one pre-training seed; probe seeds vary only random projections. The bootstrap cannot establish training-level ranking stability, and nonsignificance does not establish equivalence. Additionally, Section 5 claims that anti-collapse regularizers fail, although Appendix E.5 explicitly states they were not tested.
Questions for authors
  1. Do the key comparisons survive matching realized mask ratios and repeating representative configurations with independent pre-training seeds?
  2. Can recommendations distinguish a useful default from a reproducible optimum, particularly given tuned random masking?
  3. Were anti-collapse regularizers tested? Otherwise, please present their predicted failure as a hypothesis.
1 comment
AuthorsOpenReview ↗

Response to Reviewer 44bh

Thank you for the careful reading. The reviewer caught a real inconsistency between Sec. 5 and App. E.5, which we fixed.

W1, Q2. Tuned random masking, default vs. optimum. We agree. In Tab. 16, random masking reaches 0.83 under MAE at ρ = 0.70, against 0.82 for (9 cm, 2 s) at its best ratio.

The recommendation against random masking is particularly relevant at fixed ρ = 0.55, and we removed the blanket statement. What Tab. 16 supports is that random masking needs ρ tuned (0.59 to 0.83 under MAE), while (9 cm, 2 s) is the least sensitive geometry (0.74 to 0.82).

Under JEPA, (9 cm, 2 s) has the higher mean at every ρ (one run per cell, so we claim no significance). We now present (9 cm, 2 s) as a robust default inside a region (L ≤ 2 s, r ∈ [6, 12] cm), where 5 of 6 cells are in the top group of each framework. Since the pretraining SD is of the order of the differences inside this region (see our answer to Cyzq, W3), we do not claim a reproducible optimum within it.

W1, Q1. Matched mask ratio. The realized ratio departs from 0.55 mainly for long blocks. For those, Tab. 16 varies ρ directly for (12 cm, 16 s): it stays at 0.57–0.63 under MAE for every ρ from 0.40 to 0.85, below (9 cm, 2 s) at every ρ, so its low score comes from the geometry.

For short blocks (L ≤ 2 s), all configurations realize 53–56%, and geometry still separates them (App. H.2):

Mask (r, L)Realized ρMAEJEPA
(9 cm, 2 s)53.4%0.8000.826
(6 cm, 2 s)56.0%0.7870.757
(12 cm, 2 s)53.8%0.7620.772
(one, 2 s)54.8%0.7220.746
(one, 1 s)55.0%0.7100.701
(all, 2 s)55.4%0.6520.054

All spherical caps at 2 s are in the top group in both frameworks. Random masking and r = all are not, at either block length. For independent pretraining seeds, (9 cm, 2 s) stays above random masking for every matched seed (36 new pretrainings, see our answer to Cyzq, W3).

W2, Q3. Claims and measured uncertainty. We agree with the three points.

  • Pretraining seeds: now measured, see above.
  • The bootstrap resamples datasets and probe seeds, so it measures evaluation variance, not training-level ranking stability. Throughout the paper, "equivalent" and "ties" become "statistically indistinguishable at our resolution", and we state that non-significance does not establish equivalence.
  • Sec. 5 now presents the failure of anti-collapse regularizers as a hypothesis, consistent with App. E.5. We then tested it with a full VICReg pretraining, which did not prevent the collapse (see our answer to 9MK7, W2). C-JEPA remains untested, and the text says so.
Reviewer_9MK7OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper systematically investigates the impact of spatio-temporal masking geometry on EEG foundation models. The authors introduce a unified masking framework parameterized by spatial radius and temporal length, and conduct a controlled evaluation across two self-supervised learning frameworks, MAE and JEPA. The paper also identifies and analyzes a JEPA-specific failure mode, termed bias-inflation collapse, which is not detected by conventional collapse diagnostics.

Critical strengths
  1. This paper systematically formalizes the masking strategy with training various model across two SSL framework. The experimental evaluation is extensive, covering 58 pre-trained models across 29 masking configurations and two representative self-supervised learning frameworks, MAE and JEPA.
  2. The identification and analysis of bias-inflation collapse in JEPA is an interesting contribution. The authors go beyond reporting downstream performance degradation by investigating the underlying representation statistics and explaining why conventional collapse diagnostics may fail to detect this behavior.
Critical weaknesses
  1. The paper identifies bias-inflation collapse but offers no mitigation beyond avoiding problematic masks. Could modifying positional encoding or the JEPA prediction objective prevent this collapse, thereby validating the proposed mechanism?
  2. The observed sensitivity of JEPA to masking geometry may be specific to the unregularized JEPA implementation. It remains unclear whether similar failure modes would occur in JEPA variants with explicit anti-collapse mechanisms.
Questions for authors

Please see weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 9MK7

Thank you for the positive review and for the two questions on mitigation, which led to new experiments.

W2. Anti-collapse regularizers. We pretrained JEPA (all, 2 s) with VICReg applied to the encoder output tokens. VICReg did not prevent the collapse.

From epoch 2 onward, the representations had enough variation to meet VICReg's variance requirement. Yet EEG signal-related variation dropped from 1.01 to 0.09, while position-related variation remained above 1.

The final score was −0.244, compared with 0.054 without VICReg and 0.826 for (9 cm, 2 s) (scores are min–max normalized per dataset over the 58 grid runs; see App. C.4, so a run outside the grid can fall below 0).

This is what the mechanism predicts: a variance computed over all tokens can be satisfied by the position-dependent mean alone. One run at one weighting does not cover every regularized variant. The revision states both.

W1. Mitigation and mechanism.

The practical mitigation is the mask itself: the collapse occurs when the mask covers all channels simultaneously, and never within the recommended region. To probe the mechanism, we ran short interventions on JEPA (all, 2 s), each continuing the epoch-3 checkpoint for 300 steps and evaluated on 3 datasets (App. E.6):

  • Changes to the positional encoding (encoder positional dropout, predictor query jitter) or to the prediction target change the score by −1.5 to +0.7 pp. These changes leave the position-indexed solution in place: positional dropout barely reduces the positional spread (0.76 vs. 0.85), likely because the predictor still receives the query position when all same-time channels are masked.
  • Most terms that penalize low variance across recordings at each (channel, time) position, which is exactly the position-indexed solution, help: the ratio of EEG to positional spread rises from 0.33 to 0.83–1.17, and the score by 1.8 to 3.5 pp.
  • None of them reaches the unregularized (9 cm, 2 s), which stays 2.5 pp above the best arm.

These runs were single-seed, so we report them as exploratory evidence consistent with the mechanism, not as a fix compared with our masking systematic study.

Thanks for the detailed response and for conducting additional experiments to address my concerns. The full VICReg pretraining results provide convincing evidence that standard variance regularization does not prevent the observed collapse. The targeted interventions also offer useful support for the proposed mechanism.

My concerns are largely addressed. I appreciate that the revised manuscript explicitly acknowledges this limitation and avoids overgeneralizing the findings to other JEPA variants.

Although the paper does not introduce a substantial methodological advance, I think its systematic study of masking geometry in EEG foundation models and the analysis of bias-inflation collapse in JEPA are valuable contributions. I therefore maintain my rating of 3, weak accept.

Reviewer_CyzqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a controlled study of masking geometry for EEG foundation model (FM) self-supervised learning. Using a single REVE backbone, it varies spatial radius and temporal block length across 29 masking configurations under two pretraining objectives, masked autoencoding (MAE) and joint-embedding predictive architecture (JEPA), resulting in 58 experiments. The models are evaluated on 12 OpenEEGBench datasets using frozen-feature linear probes. The study finds that short temporal blocks with moderate spatial extent generally perform well for REVE.

The study is carefully executed and potentially useful to researchers applying EEG foundation models. However, its technical impact appears limited. The main contribution is a pipeline-specific empirical map of masking geometries obtained using a single backbone, pretraining corpus, tokenization scheme, and evaluation protocol. Although the results provide useful engineering guidance, they do not establish a broadly transferable principle or constitute a substantial methodological contribution.

Critical strengths
  • The detailed analysis of masking strategies under both MAE and JEPA makes the qualitative conclusions more convincing.

  • The study evaluates the models across 12 datasets.

  • The computational costs, implementation details, ablations, and limitations are reported thoroughly.

Critical weaknesses
  1. The practical significance of pretraining-mask optimization remains unclear

The paper provides a detailed investigation of how masking geometry affects frozen-feature representations. However, the practical importance of this optimization axis is not sufficiently established.

For practitioners, downstream adaptation, particularly fine-tuning the pretrained model on task-specific data, may be more consequential than optimizing the pretraining masking geometry alone. A configuration that performs best under frozen-feature linear probing does not necessarily produce the best results after full or partial fine-tuning. Fine-tuning may alter the relative ranking of the pretrained representations, potentially reducing or amplifying the differences observed in this study.

The paper would therefore benefit from evaluating whether the identified masking preferences persist under realistic downstream adaptation settings, including full fine-tuning or parameter-efficient fine-tuning. Comparisons of downstream performance, data efficiency, and computational cost would help establish whether selecting a particular masking geometry yields meaningful practical benefits beyond those available through standard adaptation.

  1. The computational-efficiency claim is not clearly attributable to masking geometry

The claim of achieving “REVE-Base-level performance for a fraction of its compute” appears less directly related to the paper’s main contribution. The computational savings seem to arise primarily from other aspects of the training pipeline, including model size, dataset size, and training schedule, rather than from the optimized masking strategy itself.

Furthermore, the paper reports comparable performance across a broad range of masking configurations. This raises the question of whether the recommended geometry contributes meaningfully to the reported cost savings, or whether similar performance could be obtained with other configurations at comparable computational cost.

The authors should distinguish savings attributable to the overall training pipeline from those attributable specifically to masking geometry and substantiate the latter if computational efficiency is presented as a contribution.

  1. Single-seed pretraining variability

Each masking configuration appears to be represented by a single pretrained checkpoint. The five downstream seeds vary only the projection head or linear probe. Consequently, the reported variability does not capture the potentially substantial variance arising from pretraining itself. This makes it difficult to determine whether differences among closely performing masking configurations are robust.

Questions for authors

-Were any configurations pretrained more than once, and how large was the checkpoint-level variability in any pilot runs?

-How stable is the selected geometry under leave-one-dataset-out selection?

-Does the preferred masking geometry remain competitive under full or parameter-efficient fine-tuning, and does it improve downstream data efficiency or computational cost?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Cyzq

Thank you for the detailed review and the precise questions. We address them in order.

Transferability & methodological contribution. To address the point about broader transferability, we conducted a cross-backbone ablation study. We found out that the recommendation is not specific: retrained with (9 cm, 2 s) instead of their published random mask, both CBraMod improves from 0.597 to 0.640 and LaBraM from 0.605 to 0.642 (App. H.1).

To put this in perspective, switching to our recommended masking geometry produces a larger boost (+3.7 to +4.3 pp) than the replicate step-up between successive generations of EEG foundation models (App. G).

Beyond the best mask, we identify which masks fail and why: masking all channels at the same time underperforms under MAE and collapses under JEPA, through a mechanism that standard collapse diagnostics miss (App. E).

The good mask is never worse, and a bad mask is not fully rescued by fine-tuning.

W1, Q3. Fine-tuning.

The paper asks how the mask shapes the pretrained representation, and a frozen probe measures it without an adaptation recipe that interacts with the backbone. Linear probing of frozen features is the standard protocol to assess self-supervised representations (e.g., SimCLR, DINO, I-JEPA).

We agree that fine-tuning is the other relevant setting, so we fine-tuned 6 checkpoints on 3 datasets with rank-16 LoRA and full fine-tuning (3 seeds, fixed recipe not tuned per checkpoint, App. H.6). Balanced accuracy under JEPA:

JEPAProbeLoRAFull FT
bcic2a, (9 cm, 2 s)0.4600.4580.444
bcic2a, (all, 2 s)0.2590.4130.392
arithmetic, (9 cm, 2 s)0.7290.6930.657
arithmetic, (all, 2 s)0.5980.5640.528
MDD, (9 cm, 2 s)0.8550.8480.853
MDD, (all, 2 s)0.6990.8450.860
  • (9 cm, 2 s) is in the top group of its framework in all 12 settings (2 frameworks × 3 datasets × 2 protocols).
  • Adaptation reduces the gaps, most on the clinical dataset. With 3 seeds, the collapsed JEPA (all, 2 s) remains separable from (9 cm, 2 s) only under full fine-tuning on bcic2a (5 pp below).
  • Data efficiency. With 10% of the labels on arithmetic (about 120 trials), the JEPA (9 cm, 2 s) probe reaches 0.697, against 0.539 for ShallowFBCSPNet and 0.523 for EEGNet trained from scratch. Under JEPA, (9 cm, 2 s) has the best mean at every label fraction on arithmetic. On bcic2a, the top masks are indistinguishable below 100% of the labels.

So the choice of mask matters most for frozen use, and the recommended mask stays in the top group under every protocol.

W2. Compute claim. We agree. The saving comes from the pipeline: a smaller model (12.7M vs. 69M parameters), less data (4.4 vs. 6 TB) and a shorter schedule. Other configurations of the top group reach the same level at the same cost.

The role of the mask is to avoid the failure modes, such as JEPA at r = all, which fall well short at the same budget. Finding 4 said this, but the abstract, the introduction (l. 74–75), Sec. 4 (l. 340–342) and Sec. 5 (l. 431–432) could be read otherwise. The abstract now reads: "Our compact pipeline reaches REVE-level downstream performance at a fraction of REVE's pre-training compute, provided the mask avoids the failure modes we identify." The other three sentences are aligned with it.

W3, Q1. Pretraining seeds. The reviewer is right: each configuration was pretrained once, and the bootstrap captured evaluation variance only. We ran 36 full pretrainings with the grid recipe, along two axes (5 initialization seeds, 5 data-order seeds), for (9 cm, 2 s) and random masking (one, 1 s). Normalized 12-dataset score, mean ± SD over 5 pretrainings:

Seed axis(9 cm, 2 s)RandomGap / pooled SD
MAEInitialization0.791 ± 0.0180.713 ± 0.0085.5
MAEData order0.793 ± 0.0210.716 ± 0.0104.6
JEPAInitialization0.753 ± 0.0420.668 ± 0.0941.2
JEPAData order0.778 ± 0.0120.712 ± 0.0392.3

Our finding about the (9 cm, 2 s) is above random masking for every matched seed, on both axes and in both frameworks. JEPA is more variable, mostly because of one random-masking run at 0.507. The pretraining SD of (9 cm, 2 s) (0.01 to 0.04) is of the same order as the differences inside the top group, so the grid identifies a region rather than a single optimum (see our answer to 44bh, Q2).

Q2. Leave-one-dataset-out. The selection is stable. Re-running the paper's two-level bootstrap (Sec. 3.2, Fig. 3B) on each subset, (9 cm, 2 s) stays in the top group in 12/12 LODO and 66/66 leave-two-out subsets, in both frameworks (App. H.5). It is the winner in 9/12 (MAE) and 11/12 (JEPA) LODO subsets.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: new experiments and clarified claims

We thank the reviewers for their reviews.

They find the study systematic and carefully executed, with 58 pretrained models across 29 geometries and two SSL frameworks (Cyzq, 9MK7, 44bh), and the analysis of bias-inflation collapse an interesting contribution that standard collapse diagnostics miss (9MK7, 44bh). The masking geometric findings are robust across multiple EEG foundation models, e.g., REVE, CBraMod, and LaBraM.

The open questions were about pretraining variance, the separation of geometry from mask ratio, fine-tuning, and anti-collapse regularizers. We ran the following experiments, now in App. E.6 and H.2, H.4–H.6 of the revised PDF (changes in red):

  • 36 additional full pretrainings: 5 weights initialization seeds and 5 data-order seeds, for (9 cm, 2 s) and random masking, under MAE and JEPA (Cyzq, 44bh).
  • LoRA and full fine-tuning of 6 checkpoints on 3 datasets, and low-label probes against supervised CNNs (Cyzq).
  • Leave-one- and leave-two-datasets-out selection (Cyzq).
  • Geometries compared at matched realized mask ratio (44bh).
  • A full JEPA pretraining with VICReg at r = all, and short interventions on the positional encoding and the objective (9MK7, 44bh).

The main conclusions hold. (9 cm, 2 s) beats random masking for every matched pretraining seed, stays in the top group in 12/12 LODO and 66/66 L2DO subsets, and stays in the top group under LoRA and full fine-tuning. In one full pre-training at fixed weights, VICReg does not prevent bias-inflation collapse.

Following the reviewers, we also made four claims more precise:

  1. The compute saving comes from the pipeline. The mask only avoids the failure modes that fall well short of REVE-Base at the same budget (Cyzq).
  2. Tuned random masking reaches the recommended geometry under MAE. We now present (9 cm, 2 s) as a robust default inside a region (L ≤ 2 s, r ∈ [6, 12] cm), not as a unique optimum (44bh).
  3. Section 5 now states, consistently with App. E.5, that the failure of anti-collapse regularizers is a hypothesis, and reports the VICReg run that tests it (44bh).
  4. "Equivalent" and "ties" are replaced by "statistically indistinguishable", and we state that non-significance does not establish equivalence (44bh).

To our knowledge, this is the first study of masking geometry for EEG foundation models where everything else is fixed.

Beyond a map for one pipeline, it gives a default that also helps other architectures: retrained under MAE, CBraMod and LaBraM score higher with (9 cm, 2 s) than with their published random mask (0.640 vs. 0.597, 0.642 vs. 0.605; App. H.1), and a JEPA failure mode with a mechanism that standard diagnostics do not detect. We are ready to run further analyses during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.