acceptodds
Under review as a conference paper at ICLR 2027

When Masked Reconstruction Becomes Spatial Interpolation: What EEG Foundation Models Miss

Abstract

EEG foundation models pretrained by masked reconstruction often trail smaller supervised models, and their gains may depend on task-specific heads or fine-tuning rather than on the pretrained representation. This paper investigates what masked reconstruction transfers to downstream EEG tasks, and why it behaves differently from images. A standard criss-cross Transformer is pretrained on 5,200 hours of EEG data and evaluated on seven benchmarks under linear probing and fine-tuning with a standardised head. When predicting masked EEG patches, a linear map from the visible electrodes at the same time step recovers 66–84% of the encoder's improvement over a simple predictor, so the pretext task can largely be solved by spatial interpolation. Accordingly, during pretraining, spatial attention rapidly converges to a largely trial-invariant map concentrated on neighbouring electrodes, which the same architecture trained on task labels does not form. In contrast, ViT-MAE on images exhibits content-dependent attention under the same analysis. On BCI IV-2a, residual-stream probes show high subject decodability (0.93) but weak class decodability (0.30); across datasets, per-electrode band power is strongly represented before mean pooling but partly lost by it, while channel covariance is substantially less accessible. In a released EEG foundation model, 99% of the pooled representation's energy lies in a component shared across trials, comparable to random initialisation, and falls to 30% after fine-tuning. Replacing learned spatial attention with fixed maps has little effect on downstream performance on most benchmarks, whereas adding covariance and differential entropy features improves performance by up to 11 percentage points on per-subject motor imagery. Together, these results diagnose a modality-specific spatial shortcut in masked EEG pretraining and motivate objectives that explicitly target input-dependent cross-channel structure.

Primary area:
foundation or frontier models, including LLMs
Keywords:
EEG foundation models, masked reconstruction, spatial attention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WnESOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether EEG foundation models pretrained with masked reconstruction rely primarily on spatial interpolation between electrodes rather than learning useful EEG representations. The authors propose the Spatial Short-cut hypothesis and analyze reconstruction behavior, spatial attention, and learned representations. Their experiments reveal highly trial-invariant spatial attention across several models and examine its implications through downstream interventions. Overall, the paper addresses a valuable research question and provides useful empirical observations. However, the interpretation of trial-invariant spatial attention and its relationship with representation quality remain insufficiently supported.

Critical strengths
  1. Addresses an important question about the limitations of masked reconstruction in EEG foundation models.

  2. Provides a systematic analysis of reconstruction behavior, spatial attention, and learned representations.

  3. Evaluates multiple EEG foundation models and datasets, with additional controls involving initialization, supervised training, and masking strategies.

  4. Connects the findings to downstream performance through attention substitution, feature augmentation, and fine-tuning experiments.

Critical weaknesses
  1. The presentation lacks clarity, particularly regarding the objectives, interpretations, and connections between experiments.
  2. The interpretation of trial-invariant spatial attention is insufficient. Such patterns may reflect meaningful EEG spatial structure, and their causal relationship with representation limitations remains unclear.
  3. Some diagnostic experiments have limited coverage across models and datasets, with insufficient analysis of subject-level and trial-level variability.
  4. The representation analysis relies mainly on DE, covariance, and linear regression, which may not fully characterize the learned EEG information.
  5. The paper identifies potential limitations of masked reconstruction but provides limited discussion of concrete and feasible directions for addressing them.
Questions for authors
  1. Given the inherent spatial structure of EEG, how do the authors distinguish meaningful spatial priors from potentially restrictive trial-invariant spatial maps? Do spatial attention patterns differ across downstream tasks?
  2. In Figure 2, both Static Share and downstream performance increase during early training. Does this suggest that higher Static Share does not necessarily indicate limited representation quality?
  3. In Section 5.1, Subject Identity is more decodable than Task Class, but this may reflect inherent individual differences in EEG. Is this sufficient evidence of limited task-relevant representations? Additionally, does the comparison in Figure 3(b) account for correlations among trials from the same subject?
  4. Why are only DE and covariance used to evaluate neurophysiological information? Would additional EEG features or nonlinear regression models lead to different conclusions?
  5. Could the authors compare the final spatial attention maps of pretrained-and-fine-tuned models and models trained directly under supervision, particularly their Static Share and spatial distributions?
6 comments
AuthorsOpenReview ↗

Question 1 response

Question 1: Given the inherent spatial structure of EEG, how do the authors distinguish meaningful spatial priors from potentially restrictive trial-invariant spatial maps? Do spatial attention patterns differ across downstream tasks?

Response: Thank you for your time reviewing our paper and for raising this point. A spatial attention map that stays similar across trials can still reflect genuine EEG structure, and we do not regard it as a flaw in itself. We take a spatial prior to be useful for a downstream task if it is specific to that task and if the transferred representation depends on it. The pretrained map meets neither condition.

Same map across datasets. The pretrained encoder uses a similar pattern for combining electrode signals on every dataset. On their shared electrodes, its average maps correlate at Pearson r = 0.91–0.96 across datasets (permutation null ≤ 0.23), despite different subjects, montages and tasks. The same holds for LaBraM (r = 0.91–0.99) and ST-EEGFormer (r = 0.89–0.97; permutation null ≤ 0.27). The maps of the same architecture trained with supervision differ from one dataset to another (r = 0.16–0.24, no higher than the permutation null). This is added to Section 4.1, and the calculation is detailed in the new Appendix H.4 (Cross-Dataset Comparison).

Performance does not depend on the learned map. We ran two controls. First, we left spatial attention untrained during pretraining and compared the resulting encoder with the original run at the same epoch. Second, we shuffled which electrode signal went into each input position of the frozen encoder and trained a new linear classifier on its outputs.

Table X1. Frozen linear-probe balanced accuracy (%, mean ± standard deviation over three seeds) for the spatial attention controls. (i) The two runs are compared at the same pretraining epoch (epoch 9); from Table 16, Appendix L.3. (ii) The encoder is held fixed and a linear classifier is fitted separately for each input ordering; from Table 23, Appendix Q.3.

ControlConfigurationISRUCFACEDMA
(i) Spatial attention during pretrainingLearned (original run)66.9 ± 1.722.2 ± 0.573.7 ± 7.5
Untrained (frozen at initialisation)64.7 ± 3.222.7 ± 0.867.6 ± 5.0
(ii) Electrode order at the inputIntact69.0 ± 1.121.8 ± 0.970.3 ± 7.6
Permuted66.8 ± 3.421.3 ± 1.071.3 ± 4.5

The trial-invariant map is thus a prior about electrode proximity, identical across tasks, that the transferred representation does not rely on. It can also be improved on where a task needs other cross-channel relationships: on FACED, replacing the frozen encoder's maps by those of a model trained on the task, with all pretrained weights unchanged, raises balanced accuracy from 21.8 to 30.5, and to 40.2 with maps taken from a fine-tuned model. Together, this indicates that the learned spatial map is not what transfers, and that task-specific routing can add to it.

We added these configurations (untrained map, permuted electrodes, supervised map swap) to Section 5.2 and Figure 4, and moved the previous downstream interventions to Appendix Q.1. The runs with untrained spatial attention are reported in Appendix L.3 (Table 16), the map swap in Appendix Q.2 (Table 22), and the electrode-permutation results for the released models in Appendix Q.3 (Table 23).

AuthorsOpenReview ↗

Question 2 response

Question 2: In Figure 2, both Static Share and downstream performance increase during early training. Does this suggest that higher Static Share does not necessarily indicate limited representation quality?

Response: Thank you for highlighting this. A higher Static Share means that the spatial attention maps are more similar across trials. In Figure 2, Static Share and downstream linear-probe accuracy both increase during early pretraining. As you point out, their joint rise leaves open whether learning the spatial attention weights was necessary for the performance gains.

To test this, we repeated pretraining with the spatial-attention parameters frozen, either from initialisation or after epoch 1, while the rest of the encoder continued to train. We evaluated both variants at epoch 9 with the same linear-probe protocol on the frozen encoder. The results are in the new Appendix L.3 (Freezing spatial attention in training), Table 16.

Table 16. (in the paper, Appendix L.3) Linear-probe balanced accuracy (%, mean ± standard deviation over three seeds) with spatial attention trained or frozen during pretraining.

EncoderISRUCFACEDMA
Original run, epoch 161.5 ± 3.518.9 ± 1.269.3 ± 8.5
Original run, epoch 966.9 ± 1.722.2 ± 0.573.7 ± 7.5
Spatial attention frozen from initialisation, epoch 964.7 ± 3.222.7 ± 0.867.6 ± 5.0
Spatial attention frozen from epoch 1, epoch 966.8 ± 0.921.8 ± 1.373.1 ± 8.5

Freezing spatial attention after epoch 1 gives scores close to those of the original run at epoch 9. Freezing it from initialisation, where attention is close to uniform and Static Share is near zero, gives comparable scores, within the spread across seeds. Downstream gains therefore occur while the spatial-attention parameters are held fixed, and with either a high or a near-zero Static Share: the two are not coupled.

A likely explanation for the gains is that pretraining makes per-electrode spectral information more linearly accessible. The decodability of differential entropy (DE) from the pooled representation is higher after pretraining than at initialisation for our encoder, ST-EEGFormer and REVE on all five datasets tested (Appendix I.2).

Table 12. (in the paper, Appendix I.2) Decodability of per-electrode DE (R² on held-out subjects) from the pooled last-layer representation, at initialisation → after pretraining.

ModelFACEDBCI IV-2aMumtazMAISRUC
Ours0.42 → 0.680.64 → 0.860.44 → 0.730.22 → 0.520.56 → 0.85
ST-EEGFormer0.45 → 0.640.65 → 0.780.49 → 0.670.29 → 0.420.61 → 0.81
REVE0.23 → 0.580.00 → 0.740.00 → 0.600.00 → 0.290.09 → 0.77

We agree that a higher Static Share does not by itself indicate poorer representation quality. It characterises the spatial solution that masked reconstruction settles on, and we now describe it as a by-product of solving the objective rather than a driver of transfer. This is stated in Section 4.2, after the discussion of the training trajectory. These values are reported in Appendix I.2, where the initialisation columns for ST-EEGFormer and REVE have been added alongside those of our encoder.

AuthorsOpenReview ↗

Question 3 response

Question 3: In Section 5.1, Subject Identity is more decodable than Task Class, but this may reflect inherent individual differences in EEG. Is this sufficient evidence of limited task-relevant representations? Additionally, does the comparison in Figure 3(b) account for correlations among trials from the same subject?

Response: Thank you for raising both points. We agree that high subject decodability alone is insufficient evidence of limited task-relevant information. Hand-computed differential entropy and covariance features alone identify participants with balanced accuracy of 0.78–0.99 across MA, BCI IV-2a and ISRUC, which illustrates how strongly EEG reflects individual differences.

Does subject-specific variation make task decoding harder? To test this directly, we asked whether subtracting each participant's average representation makes task labels easier for a linear classifier to decode. For each test participant, we estimated the mean representation from half of their windows, without labels, and subtracted it from the representations of the remaining windows. Centred and uncentred performance are compared on exactly the same evaluation windows, over five subject-disjoint folds repeated three times.

Table 21 (in the paper, Appendix O.2). Balanced accuracy of a linear readout shared across subjects, before and after subtracting each test subject's mean representation, estimated on separate unlabelled calibration windows of that subject.

ModelReadoutMABCI IV-2aISRUC
OursShared linear readout0.690.390.70
After label-free centring0.800.460.72
ST-EEGFormerShared linear readout0.690.360.70
After label-free centring0.760.430.69
LaBraMShared linear readout0.690.290.70
After label-free centring0.770.300.69

Centring improves balanced accuracy in seven of nine comparisons, by 1.5 to 11 percentage points. All three encoders improve on MA and BCI IV-2a, while ISRUC shows mixed results. This suggests that subject-specific shifts in the frozen representations can make task decoding harder for a linear classifier in some settings. We state this in Section 5.1, under "Subject identity is more decodable than task content", and give the full analysis in Appendix O.2 (Table 21).

Correlations among trials from the same subject. The original Static Share estimate pooled all windows. To check whether repeated windows from the same participants drove the result, we recomputed Static Share with one randomly selected map per participant, repeating the draw 20 times. The resulting estimates (0.86–0.88) are close to the pooled estimate on the same sample (0.86–0.87). Static Share is also high within participants (0.90–0.92) and across participant-average maps (0.94–0.95), and each participant's average map correlates strongly with the average map of the remaining participants (Pearson r ≥ 0.85, 0.96–0.97 on average). Together, these checks show that the spatial pattern remains the same when each participant contributes equally.

We added this result to Section 4.1 and describe the participant-level checks in Appendix H.5 (Participant Identity and Static Share, Table 9). The figure itself (Figure 3b of the original submission) is now Figure 9b in Appendix N.

Reviewer_LDLbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether masked EEG pretraining relies on spatial interpolation and examines the implications for learned representations. It finds that fixed spatial predictors recover much of the reconstruction gain and that the analyzed spatial attention patterns are largely invariant across trials in the tested models. Representation probes and downstream experiments further reveal stronger subject than task decodability, limited linear accessibility of covariance on some datasets, and task-dependent benefits from adding covariance or differential entropy features.

Critical strengths

The paper combines reconstruction baselines, attention diagnostics, and downstream interventions to provide a useful empirical investigation of spatial shortcuts in EEG foundation models.

Critical weaknesses
  1. The main contribution is not fully established. The experiments identify spatial predictability, static attention, and weak transfer, but do not demonstrate that the proposed shortcut causes the downstream limitations. A controlled experiment connecting shortcut reduction to improved transfer is needed.

  2. The model coverage is too limited to support the broad conclusions. The evaluation omits recent approaches such as REVE, CodeBrain, and CSBrain. Without evaluating representative alternatives, it remains unclear whether the spatial shortcut is a general limitation of masked EEG pretraining or a behavior specific to the selected models.

Questions for authors
  1. The performance of the paper is not clear. it would be nice to have a main figure to help with the understanding.

  2. Why do the attention maps in figure 3a look similar? does this mean that the spatial attention fail to distribute to different region given a variaty of class label?

2 comments
AuthorsOpenReview ↗

Question 1 response

Question 1: The performance of the paper is not clear. It would be nice to have a main figure to help with the understanding.

Response: Thank you for your time reviewing our paper. We added an overview figure (Figure 1), referenced at the start of Section 4, which links the three steps of the argument to the sections that test them. The diagnostic measures are now defined together in Section 3.3, and each section opens by stating what it tests.

Figure 1: Overview. (a) A masked channel can be reconstructed from neighbouring electrodes, whereas a downstream task may also depend on relations between distant electrodes. (b) Mean spatial attention of four pretrained models on the electrodes of MA; the maps are similar across models and concentrated on nearby electrodes. (c) The learned map can be replaced. Learned: the encoder is pretrained with learned spatial attention. Fixed kernel: the same encoder is pretrained from scratch with a fixed scalp-distance kernel, so no spatial map is learned. Downstream balanced accuracy is comparable.

W1 (a model not trained through the shortcut). We ran a controlled intervention that gives the pretrained encoder spatial attention that was not learned through the shortcut. In the frozen pretrained encoder, only the spatial attention maps are replaced, window by window, by those that a supervised or a fine-tuned model of identical architecture produces on the same input. All pretrained weights are unchanged; only the spatial mixing coefficients differ, and a linear probe is fitted as usual.

Table X1. Linear-probe accuracy (%) of the frozen pretrained encoder with its own spatial maps and with maps from a task-trained model of the same architecture. Balanced accuracy for FACED and ISRUC; accuracy for BCI IV-2a. The values for our encoder are from Table 22 (Appendix Q.2), and the ST-EEGFormer values with its own and with fine-tuned maps are from the text of Section 5.2. The remaining ST-EEGFormer value and the last two rows, which give the accuracy of the task-trained models themselves, are additional results reported here.

Spatial maps of the frozen encoderOurs, FACEDOurs, ISRUCST-EEGFormer, BCI IV-2a
Its own (pretrained)21.869.040.6
From the supervised model30.571.442.2
From the fine-tuned model40.274.455.2
Supervised model itself49.471.032.3
Fine-tuned model itself53.373.256.0

With the maps of the fine-tuned model, accuracy rises from 21.8 to 40.2 on FACED and from 40.6 to 55.2 on BCI IV-2a, which recovers 58% and 95% of the gap to the fine-tuned model; on ISRUC the frozen encoder with these maps reaches the fine-tuned model's accuracy. The maps of the supervised model also help on FACED (30.5) and ISRUC (71.4). On BCI IV-2a the supervised model is itself below the frozen encoder, probably because ST-EEGFormer is large for such a small dataset, and its maps bring little (42.2).

The same pretrained weights therefore perform better once given the maps of a model trained on the task (on FACED, Static Share 0.25–0.33) in place of their own nearly fixed, distance-shaped map (0.88). The pretrained map is sufficient for reconstruction, but it does not supply the routing the task can use. This is reported in Section 5.2 and Figure 4, with the procedure in Appendix P and the results in Appendix Q.2.

W2 (additional models). REVE and CSBrain are now included in the attention diagnostic (Table 2, Section 4.1). REVE (masked autoencoding) shows the same static, distance-shaped map as the other masked models: Static Share 0.91–0.94, distance R² 0.72–0.83 (permutation null ≤ 0.05). CSBrain restricts spatial attention to small fixed groups of electrodes, one from each brain region; on these its Static Share is 0.70–0.73, with R² at the null level, since the allowed electrodes come from different regions. In CodeBrain, attention is limited to each token itself, so all spatial mixing comes from a convolution whose kernel does not depend on the input, which makes it static by construction; we therefore discuss it here and do not report a measured value. In every measured case, spatial routing after masked pretraining is far more static than in the same architecture trained on the task (0.04–0.25).

AuthorsOpenReview ↗

Question 2 response

Question 2: Why do the attention maps in Figure 3a look similar? Does this mean that the spatial attention fails to distribute to different regions given a variety of class labels?

Response: Thank you very much for giving us a chance to clarify this. The maps look similar because the pretrained encoder gives nearly the same weights to the electrodes whose signals are combined at F4, on all three example trials, although they come from different participants and classes. The figure (which is now placed in the Appendix N (Figure 9a)) illustrates a pattern that holds in general: the dominant spatial pattern is shared across inputs.

The quantitative analysis confirms this. Across the datasets examined, Static Share is 0.87–0.90: 87–90% of the non-uniform attention energy is the same on every trial. This statistic pools trials of all classes, so at most 10–13% of that energy varies with the input, class included. Inter-electrode distance explains 70–87% of the variation in the row-centred mean attention logits (see Table 2 ).

So yes: after masked pretraining, spatial attention largely does not redistribute according to the class. This is a property of the pretrained weights and not of the architecture. Trained on the task labels, the same architecture learns input-dependent maps (Static Share 0.04–0.25), and replacing the frozen encoder's maps by those of a task-trained model raises accuracy (FACED: 21.8 to 30.5 with supervised maps and 40.2 with fine-tuned maps; see our response on W1). Similar attention weights can still produce different outputs, because the signals being combined vary from trial to trial; the weights that mix electrodes are what is shared.

To avoid this confusion, we made the description of the figure more precise. The comparison with vision is now in Appendix N , where the figure is Figure 9a , and its conclusion is summarised in Section 4.1 ("What produces the map").

Reviewer_q5tAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether reconstruction-based masked pretraining allows EEG foundation models to rely on spatial interpolation rather than learning input-dependent cross-channel structure.

The authors hypothesize that strong spatial redundancy can make masked EEG prediction largely solvable through fixed or trial-invariant channel mixing. Spatial redundancy, partly arising from volume conduction, is widely known in EEG. Overall, the results support a spatial-shortcut hypothesis in the tested setting, but do not yet establish a general diagnosis of masked EEG pretraining. It remains unclear whether the observed behavior is caused specifically by masked pretraining or primarily reflects the inherent geometry and spatial redundancy of EEG signals.

Critical strengths
  • The study spans multiple datasets and released foundation models, with substantial methodological detail provided in the appendices.

  • The distinction between per-electrode and pooled representations identifies a concrete downstream design issue: information accessible at individual electrodes may become less accessible after channel pooling.

Critical weaknesses
  • Strong spatial redundancy between EEG electrodes, which may be caused by volume conduction, is already well established. The paper provides useful evidence connecting this phenomenon to masked reconstruction, but it should clarify more precisely what is scientifically novel beyond demonstrating that existing EEG foundation models exploit this known redundancy. It remains unclear whether the observed behavior is caused specifically by masked pretraining or primarily reflects the inherent geometry of EEG signals.

  • The primary model uses independent random token masking at a fixed 60% masking ratio. Appendix M adds a correlation-aware masking experiment, which is a useful robustness check, but the analysis still covers a limited range of masking strategies and ratios. It remains unclear whether the conclusions hold under substantially different spatial and temporal masking geometries.

  • Nearly trial-invariant attention weights do not imply that the resulting computation is trial-invariant. The value vectors being mixed can remain strongly input-dependent, while residual connections, temporal attention, and MLP blocks may carry substantial cross-channel or task-relevant information. Conversely, variability in attention weights does not necessarily imply causal importance. The fixed-map substitution experiment is useful, but it evaluates robustness after the model has already been trained. It does not establish that a model pretrained with fixed spatial attention would learn the same representations or achieve the same transfer performance.

  • Covariance is a quadratic function of the signal. The failure of a linear probe to reconstruct covariance does not demonstrate that covariance information is absent from the learned representation. It shows only that the information is not accessible to the tested linear readout. Similarly, adding covariance or differential-entropy features supplies explicit nonlinear statistics of the raw signal while increasing the input dimensionality. Improved performance after this augmentation is not sufficient to prove that the encoder lacks the underlying information.

Questions for authors
  • Has the primary architecture been pretrained from the outset using fixed, distance-based spatial attention or without the spatial branch? If so, how does its transfer performance compare?

  • For models with joint spatiotemporal attention, how much of the full attention mass is assigned to the analyzed same-time channel block, and how much does this mass vary across inputs?

  • Do the principal findings hold across substantially different masking ratios and spatiotemporal masking geometries?

4 comments
AuthorsOpenReview ↗

Question 1 response

Question 1 (part 1 of 2): Has the primary architecture been pretrained from the outset using fixed, distance-based spatial attention or without the spatial branch? If so, how does its transfer performance compare?

Response: We thank the reviewer for this question. The additional pretraining controls address Question 1 and the concerns raised in W1 and W3. Masking robustness (W2) is partially addressed here and specifically in our response to Question 3. This comment covers Question 1 and W1; W3 and W4 follow in part 2.

To distinguish the contribution of cross-electrode mixing from that of learned spatial-routing weights, we pretrained two additional versions of the primary encoder from scratch: one with fixed distance-based spatial attention and one without the spatial branch. Both used the same pretraining data, masking scheme and training schedule as the original encoder. Transfer was evaluated with frozen encoders and linear classifiers. Unlike the attention-substitution experiment in the submitted manuscript, these controls modify the architecture from the beginning of pretraining.

Table 19 (Appendix M). Linear-probe balanced accuracy (%, mean ± standard deviation over three seeds).

EncoderISRUCFACEDMA
Primary (learned spatial attention)69.0 ± 1.121.8 ± 0.970.3 ± 7.6
Fixed distance kernel67.0 ± 3.522.6 ± 0.969.4 ± 7.9
No spatial branch20.0 ± 0.020.7 ± 1.056.1 ± 4.4
Untrained (initialisation)57.9 ± 0.418.8 ± 0.355.2 ± 8.0
  1. Fixed distance-based attention. Replacing learned spatial attention by a fixed scalp-distance kernel throughout pretraining gives downstream balanced accuracy comparable to the original encoder on the three datasets.
  2. No spatial branch. In the criss-cross encoder, the spatial branch is the only operation that directly mixes information across electrodes. Without it, the model's reconstruction error stays close to that of predicting zero for every masked patch. Its linear-probe accuracy is at the level of the untrained encoder on MA and FACED, and on ISRUC the readout predicts a single class on every seed.
  3. Electrode permutation. As an additional control, we permuted the electrode inputs of the original pretrained encoder and trained a new linear readout. Balanced accuracy changes only modestly (ISRUC: 69.0% to 66.8%; FACED: 21.8% to 21.3%; MA: 70.3% to 71.3%; Figure 4), and it stays within the spread across seeds for four released models (Table 23, Appendix Q.3). Much of the representation's linear-probe utility is therefore preserved when the original signal-to-electrode correspondence is altered.

Contribution and interpretation (W1). Our central contribution concerns the interaction between the spatial statistics of EEG and the masked-reconstruction objective, not the existence of spatial redundancy itself. As shown in Table 2, the same architecture trained with supervised labels develops substantially more input-dependent and less distance-dependent spatial attention, so EEG geometry alone does not impose static attention. Nor does the objective alone: in a masked autoencoder for images (ViT-MAE), 43% of the non-uniform attention energy is shared across inputs, against 90% for our EEG encoder (Section 4.1 and Appendix N). Under masked reconstruction on EEG, however, stable cross-electrode correlations, partly shaped by volume conduction, favour fixed spatial interpolation. The fixed-kernel control shows that learning adaptive spatial weights is not necessary for comparable downstream transfer, and the electrode-permutation control suggests that much of the measured utility does not require preserving the original electrode correspondence once the readout is refitted. These results do not imply that the learned spatial attention is harmful; they show that masked reconstruction can exploit stable spatial correlations without learning the input-adaptive spatial relationships most relevant to downstream discrimination. This redundancy was known; to our knowledge, its effect on the learned attention and on the transferred representation had not been measured.

Where this is in the revised paper: Section 5.2 ("Replacing spatial attention and splitting the representation") now presents the pretraining controls (fixed distance kernel, untrained map, permuted electrodes) in place of the earlier downstream substitution; see also Figure 4, Appendix M (Table 19) and Appendix Q.3 (Table 23).

AuthorsOpenReview ↗

Question 1 response part 2

Question 1 (part 2 of 2): W3 and W4

Trial invariance (W3). We agree that largely fixed attention weights do not imply input-invariant computation. The pretrained spatial maps are largely shared across trials (Static Share 0.87–0.90) and dominated by electrode proximity, but the value vectors and the subsequent nonlinear representations remain input-dependent. Our observation therefore concerns the learned spatial-routing weights, not the encoder's ability to represent input-dependent information. It holds for every route between electrodes: in the criss-cross architecture all cross-electrode paths pass through the spatial attention, and in the joint-attention models the attention between different electrodes at other time steps is largely fixed as well.

Table 15 (Appendix L.1). Static Share of the attention between electrodes at the same time step and at other time steps. Ranges span MA, FACED and Mumtaz.

Static ShareLaBraMST-EEGFormer
Other electrodes, same time step0.89–0.900.78–0.81
Other electrodes, other time steps0.76–0.800.82–0.86

The spatial mixing available to later layers is thus the same for every trial: which electrodes are combined, and with what weights, does not depend on the input, and later layers can only build on the combinations this fixed routing provides.

Information accessibility (W4). Both learned-attention and fixed-kernel pretraining increase the linear decodability of per-electrode differential entropy (DE), from R² = 0.65–0.70 at initialisation to 0.82–0.88 after pretraining. We focus on linear accessibility because frozen-encoder linear probing is a standard protocol for evaluating pretrained representations. We agree, however, that failure to decode covariance linearly does not establish its absence from the representation. We therefore repeated the recovery with an RBF kernel ridge regression in place of the linear probe.

Table 13 (Appendix I.3), covariance rows. Decodability (R² on held-out subjects) from the pooled representation of the pretrained encoder, linear / RBF kernel ridge. <0: worse than predicting the training mean.

FeatureFACEDBCI IV-2aMumtazMAISRUC
Covariance0.03 / 0.09<0 / <00.47 / 0.48<0 / <00.49 / 0.42
Band-limited covariance<0 / 0.01<0 / <00.23 / 0.23<0 / <00.35 / 0.29

On the datasets where the linear probe fails (BCI IV-2a, FACED, MA), the nonlinear probe recovers no more (R² ≤ 0.09). Covariance is therefore not made accessible by a more flexible readout either.

We agree about the limitation of the feature-augmentation analysis. A control that adds Gaussian noise of the same dimension shows that the gain is not due to dimensionality, but it does not rule out other explanations, so this analysis was moved to the appendix. In the main text it is replaced by a decomposition of the pretrained representation into its DE-predictable component and a remainder. On ISRUC, the DE-predictable component reaches 70.9% balanced accuracy, against 69.0% for the full representation, while the remainder reaches 37.3%; removing a random subspace or the leading principal directions with the same share of variance does not produce this drop. The same pattern holds for four released models on ISRUC, MA and Mumtaz; The same pattern holds for four released models on ISRUC, MA and Mumtaz; on FACED and BCI IV-2a the two parts do not separate. These are the datasets where DE averaged over electrodes is not sufficient for the task, since the labels depend on electrode-specific spectra that pooling removes (explained in Appendix Q.5, "Where the split separates"). These results suggest that, where the frozen encoder transfers well, much of the linear-probe utility is associated with DE-predictable spectral information, without establishing that other signal statistics are absent from the representation.

Where this is in the revised paper:

  • W3: Section 4.1 now states that trial invariance concerns the spatial attention weights while other components vary with the input, and reports the attention between different electrodes at other time steps (Appendix L.1, Table 15).
  • W4: feature augmentation is in Appendix Q.1, with the dimension control in Appendix Q.8 (Table 48). Section 5.2 presents the DE decomposition, with variance-matched controls in Appendix Q.4 (Table 24) and the released models in Appendix Q.5 (Table 25). Nonlinear feature recovery is in Appendix I.3 (Table 13).
AuthorsOpenReview ↗

Question 2 response

Question 2: For models with joint spatiotemporal attention, how much of the full attention mass is assigned to the analyzed same-time channel block, and how much does this mass vary across inputs?

Response: Thank you for this question. For each electrode–time query, we measured how much of its total attention weight goes to electrodes at the same time step. The weights are summed in the full attention map, before the same-time block is renormalised for the Static Share analysis. If attention were spread evenly over all electrode–time tokens, the same-time block would receive 1/T of the total weight, where T is the number of time steps.

Table 8 (Appendix H.1). Same-time attention mass for models with joint spatiotemporal attention. Ranges span MA, FACED and Mumtaz. Across-window standard deviations are in percentage points.

ModelTime stepsUniform baselineObserved mean massAcross-window SD
REVE520.0%46%1.3–1.5
LaBraM520.0%39–40%2.5–4.0
ST-EEGFormer402.5%17–18%1.0–1.2
BIOT911.1%11.1%0.8–1.0

Across MA, FACED and Mumtaz, REVE and LaBraM assign about twice the uniform share to same-time electrodes, and ST-EEGFormer about seven times the uniform share. In these models the fraction varies little across windows: the across-window mean accounts for 98–100% of the energy of the measured attention mass. BIOT's mean is equal to its uniform baseline, which may reflect its linear-attention mechanism, its contrastive objective or its bipolar-channel representation.

These results show that the block analysed in REVE, LaBraM and ST-EEGFormer receives a substantial and stable share of the total attention. This complements Static Share, which measures how stable the distribution of weights within the block is across inputs.

Where this is in the revised paper: the caption of Table 2 (Section 4.1) now states the attention mass assigned to the same-time block for each joint-attention model, and Appendix H.1 (Table 8) reports the full measurements and their variation across windows.

AuthorsOpenReview ↗

Question 3 response

Question 3: Do the principal findings hold across substantially different masking ratios and spatiotemporal masking geometries?

Response: Thank you for highlighting this. This response also addresses W2 (robustness to the masking strategy). The results depend on both how much EEG is hidden and which parts are hidden together. With random masking, the main attention findings hold across masking ratios from 40% to 80%. Changing the masking geometry has a larger effect on the attention maps, and the geometry that makes attention more input-dependent brings no improvement in downstream transfer.

Table 17 (Appendix M). Static Share and distance R² under different masking ratios and strategies. Ranges span MA, FACED and Mumtaz and, for random masking, all five ratios.

Masking strategyMasking ratioStatic ShareDistance R²
Random40%, 50%, 60%, 70%, 80%0.85–0.920.70–0.87
Tube60%0.86–0.910.10–0.12
Block60%0.27–0.420.11–0.19
Correlation-awaresee below0.920.67–0.75

Masking ratio. We pretrained encoders with random masking ratios of 40%, 50%, 70% and 80%, with the same training recipe as the original 60% setting. Across all five ratios, the spatial attention maps remain similar across trials and electrode distance explains much of the variation in the average map. Changing how much of the signal is visible has little effect on the tendency to use a stable, proximity-based pattern.

Masking geometry. We tested three structured patterns. Tube masking hides entire electrodes throughout each window: the maps remain similar across trials, but electrode distance explains much less of their spatial pattern. Block masking hides all electrodes during contiguous time intervals, leaving no same-time electrode signal within the masked interval: the maps vary more across trials and are weakly explained by distance. Correlation-aware masking hides each seed electrode together with its most correlated partners: the maps stay trial-invariant and distance-shaped.

Table 18 (Appendix M). Reconstruction, spectral decodability and transfer under random, block and tube masking at 60%. Balanced accuracy (%, mean ± standard deviation over three seeds) is measured with a linear probe on the frozen encoder; the block and tube encoders are evaluated at epoch 15.

MetricRandomBlockTube
Validation reconstruction MSE0.281.361.30
ISRUC DE probe (R²)0.870.780.82
ISRUC balanced accuracy69.0 ± 1.164.7 ± 2.367.1 ± 3.0
FACED balanced accuracy21.8 ± 0.922.5 ± 1.019.7 ± 1.0
MA balanced accuracy70.3 ± 7.668.1 ± 9.761.3 ± 8.9

Downstream transfer. Under block masking, the encoder makes limited progress on reconstruction: the validation MSE is 1.36, against 0.28 under random masking and 1.80 for a mean predictor. Per-electrode differential entropy is also less accessible to a linear probe on ISRUC (R² = 0.78 against 0.87), and transfer is lower on ISRUC and MA and slightly higher on FACED. Under tube masking, transfer is close to that of the original encoder on ISRUC and lower on FACED and MA. Block masking thus removes the static map by removing access to the shortcut, but the encoder then learns little and transfers no better. Lowering the Static Share does not by itself improve transfer, which suggests that the limitation lies in the masked-reconstruction objective on EEG and not in the masking pattern.

Where this is in the revised paper: Appendix M (Tables 17 and 18) gives the masking strategies and their results, together with the existing analyses of a larger model and of correlation-aware masking. The second paragraph of Section 4.2 summarises them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of responses

General response to all reviewers

We thank the three reviewers for their careful reading. Their comments led to new experiments and to a clearer line of argument. The main changes are listed below; details are in the individual responses.

1. A clearer argument. The contributions, section openings and conclusion are rewritten around three steps: masked EEG reconstruction is largely solvable with a fixed spatial map; spatial attention therefore becomes trial-invariant; and this map is a by-product of the objective, not what transfers. A new overview figure (Figure 1) links these steps to the sections.

2. Controls applied during pretraining (Section 5.2, Figure 4). Pretraining with a fixed scalp-distance kernel, or with spatial attention left untrained, gives comparable linear-probe accuracy; removing the spatial branch leaves the objective unsolved; permuting the electrodes changes accuracy only modestly (Appendices M, L.3 and Q.3; Tables 19, 16 and 23).

3. Task-trained spatial maps. Replacing only the spatial maps of the frozen encoder by those of a model trained on the task raises balanced accuracy on FACED from 21.8% to 30.5% (supervised maps) and 40.2% (fine-tuned maps) (Appendix Q.2, Table 22).

4. Broader coverage. The attention diagnostic now covers six released models, including REVE and CSBrain (Table 2), with cross-dataset map correlations and comparisons at initialisation and after supervised training for three architectures (Appendix Q.9, Tables 49 and 50). Masking ratios and geometries, the attention mass of the analysed block, and participant-level checks are reported in Appendices M, H.1 and H.5.

5. What the representation carries. The feature-augmentation analysis is replaced by a decomposition of the representation into a DE-predictable part and a remainder, with variance-matched controls and four released models (Appendices Q.4 and Q.5). Additional EEG features and a nonlinear probe confirm that per-electrode features are more accessible than relations between electrodes (Appendices I.2 and I.3).

6. Directions. The conclusion now discusses what does not work (removing access to the shortcut) and one possible direction, presented as future work.

All changes fit within the page limit. We hope the revision addresses the concerns raised.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.