- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper evaluates 29 masking geometries under MAE and JEPA using a shared EEG pre-training pipeline and frozen probes on 12 datasets. It identifies promising masking regimes and investigates a position-dominated JEPA failure mode termed bias-inflation collapse.
Critical strengths
- Systematic comparison. The shared pipeline, broad masking grid, and additional backbone and ratio experiments provide useful evidence beyond comparisons of unrelated models.
- Informative collapse analysis. Feature-mean/residual decomposition and band-power probing reveal position-dominated representations that the implemented training diagnostics miss.
Critical weaknesses
- Geometry and masking ratio are incompletely separated. Realized ratios span 50.8%–70.0% (Table 5). Appendix H.2 provides useful sensitivity checks, but random masking reaches 0.83 under MAE versus 0.82 for the recommended geometry after ratio tuning. The blanket recommendation against random masking is therefore unsupported.
- Claims exceed the measured uncertainty and interventions. Each configuration has one pre-training seed; probe seeds vary only random projections. The bootstrap cannot establish training-level ranking stability, and nonsignificance does not establish equivalence. Additionally, Section 5 claims that anti-collapse regularizers fail, although Appendix E.5 explicitly states they were not tested.
Questions for authors
- Do the key comparisons survive matching realized mask ratios and repeating representative configurations with independent pre-training seeds?
- Can recommendations distinguish a useful default from a reproducible optimum, particularly given tuned random masking?
- Were anti-collapse regularizers tested? Otherwise, please present their predicted failure as a hypothesis.
Response to Reviewer 44bh
Thank you for the careful reading. The reviewer caught a real inconsistency between Sec. 5 and App. E.5, which we fixed.
W1, Q2. Tuned random masking, default vs. optimum. We agree. In Tab. 16, random masking reaches 0.83 under MAE at ρ = 0.70, against 0.82 for (9 cm, 2 s) at its best ratio.
The recommendation against random masking is particularly relevant at fixed ρ = 0.55, and we removed the blanket statement. What Tab. 16 supports is that random masking needs ρ tuned (0.59 to 0.83 under MAE), while (9 cm, 2 s) is the least sensitive geometry (0.74 to 0.82).
Under JEPA, (9 cm, 2 s) has the higher mean at every ρ (one run per cell, so we claim no significance). We now present (9 cm, 2 s) as a robust default inside a region (L ≤ 2 s, r ∈ [6, 12] cm), where 5 of 6 cells are in the top group of each framework. Since the pretraining SD is of the order of the differences inside this region (see our answer to Cyzq, W3), we do not claim a reproducible optimum within it.
W1, Q1. Matched mask ratio. The realized ratio departs from 0.55 mainly for long blocks. For those, Tab. 16 varies ρ directly for (12 cm, 16 s): it stays at 0.57–0.63 under MAE for every ρ from 0.40 to 0.85, below (9 cm, 2 s) at every ρ, so its low score comes from the geometry.
For short blocks (L ≤ 2 s), all configurations realize 53–56%, and geometry still separates them (App. H.2):
All spherical caps at 2 s are in the top group in both frameworks. Random masking and r = all are not, at either block length. For independent pretraining seeds, (9 cm, 2 s) stays above random masking for every matched seed (36 new pretrainings, see our answer to Cyzq, W3).
W2, Q3. Claims and measured uncertainty. We agree with the three points.