- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes "Spectral Accessibility", a label-independent audit framework to evaluate what signal structures are preserved by frozen EEG foundation models (EEG-FMs). The authors construct a target state from multiresolution complex cross-spectral density (CSD) matrices, select and whiten 64 reliable PCA directions on a subject-disjoint set, and use nested cross-subject ridge regression to predict these directions from pooled model embeddings. They define SPA-20 to measure the average recoverability of the weakest 20% spectral directions under the worst input environment. Across 5 masked-reconstruction/prediction models, 7 datasets, and 4 environments (140 cells in total), all estimated linear SPA-20 scores collapse to zero, while non-zero accessibility is concentrated in upper subspaces (analyzed via a post-hoc U-SPA-20 metric).
Critical strengths
- Complementary evaluation perspective: Moving beyond downstream task accuracy to audit what intrinsic second-order spectral properties (power and cross-channel phase) are preserved in frozen representations is a well-motivated direction.
- Rigorous statistical controls and honesty: The use of split-taper reliability references and subject-block derangement nulls is solid. The authors also honestly report the complete collapse of their primary pre-specified metric (SPA-20) without disguising post-hoc upper-tail analyses as primary results.
Critical weaknesses
- Confusing feature extraction with lossless compression: The goal of EEG-FM is to extract task-relevant low-dimensional manifolds and suppress noise, not lossless signal reconstruction. The paper treats all 64 whitened PCA directions equally and penalizes models on the weakest 20% directions. Mathematically repeatable weak directions across split tapers may simply be stable physiological artifacts (e.g., EMG) or background noise. Without downstream criterion-validity experiments, treating the removal of weak spectral directions as a failure is unconvincing.
- Collapse of the main metric and degenerate theoretical bounds: L-SPA-20 is exactly 0 in all 140 test cells. At this boundary (), the derived global task-error bound degenerates into a trivial zero-state baseline, while the localized bound admits that tasks aligned with upper subspaces can still succeed. Thus, SPA-20 loses all discriminative power across models and environments, forcing the paper to rely on a post-hoc descriptor (U-SPA-20).
- Confounded pipeline due to global mean pooling: Up to 96.7% of raw target coordinates come from fine 2-second local time windows, yet the adapter pipeline crudely applies global mean pooling over all time and channel tokens before probing. Predicting local dynamic spectral blocks from a single time-averaged static vector naturally destroys degrees of freedom. Without token-level or window-aligned ablations, the audit cannot separate backbone information loss from trivial pooling loss.
- Mismatch between pretraining objectives and the linear probe: The evaluated models use very different pretraining paradigms. Counter-intuitively, LaBraM—the only model explicitly pretrained to reconstruct Fourier amplitude and phase—achieves the worst spectral accessibility score. The paper ignores that discrete vector-quantization bottlenecks may encode spectral features in non-linear manifolds (making a shared-penalty linear ridge probe unfair) and fails to explain this obvious contradiction.
Questions for authors
- Value of the lower tail: Can you provide any empirical evidence (e.g., correlation with downstream tasks) showing that preserving the weakest 20% whitened CSD directions is actually beneficial for BCI tasks, rather than just retaining task-irrelevant background variance?
- Utility of SPA-20: Given that L-SPA-20 is 0 across all 140 cells and its theoretical task bound becomes vacuous at , what is the practical utility of keeping SPA-20 as the core framework instead of directly formulating the audit around subspace alignment or effective rank?
- Unpooled ablation: Since cached embeddings are absent, can the authors run a small-scale ablation on at least one dataset (e.g., HMC) using unpooled token-level or window-aligned representations to clarify how much of the zero lower-tail is purely an artifact of global mean pooling?
- Why LaBraM fails: How do you explain that LaBraM explicitly reconstructs Fourier spectra during pretraining but ranks last in your spectral audit? Would a simple non-linear probe (e.g., 2-layer MLP) or per-coordinate ridge regularization significantly change the ordering of LaBraM?