acceptodds
Under review as a conference paper at ICLR 2027

How pretraining objectives shape EEG representations: a controlled comparison

Abstract

EEG foundation models learn transferable representations from large unlabelled recordings, but differences in datasets, architectures, and evaluation protocols obscure how pretraining objectives shape what they learn. We address this gap through a controlled comparison of masked autoencoding (MAE), autoregressive prediction (AR), and subject-level contrastive learning (SLCL), using a shared frozen tokenizer and GPT-2 backbone with pretraining data drawn from 50,894 hours of four-channel consumer EEG. Across seven downstream tasks, MAE shows the clearest performance gains on sleep staging, age prediction, gender classification, and EEG event classification, while SLCL best learns subject-specific features. Increasing pretraining data generally improves downstream performance, while LoRA fine-tuning largely preserves the relative performance of the three objectives observed under frozen probing. To interpret these objective-specific strengths, we probe how established EEG features are encoded in the learned representations. MAE retains strong band-power information from the tokenizer and correlates most with cross-channel coherence features, as it reconstructs masked tokens from visible context across channels and time. AR most clearly exposes band power trends, since it predicts subsequent activity from preceding context. SLCL favors alpha-peak frequency and alpha asymmetry, suggesting that aligning recordings from the same participant across sessions emphasizes participant-related characteristics. Layer-wise probing further reveals that task-relevant EEG information is distributed differently across model depth depending on the pretraining objective, with all seven tasks reaching their highest probe scores before the final layer. Together, these findings connect pretraining formulations to distinct spectral, spatial, and temporal feature profiles, providing an interpretable basis for understanding their downstream strengths and trade-offs.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG foundation models, self-supervised learning, pretraining objectives, representation learning, feature probing, neural signal analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hPdMOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper compares masked autoencoding, autoregressive prediction, and subject-level contrastive learning using a shared frozen EEG tokenizer and the same GPT-2 backbone architecture. Pretraining draws from a 50,894-hour private four-channel consumer-EEG corpus. Evaluation includes seven downstream tasks, LoRA adaptation, data-size comparisons, layer-wise probes, and probes of established spectral, spatial, and temporal EEG features. MAE reports the highest mean performance on six of seven downstream tasks, SLCL on cross-session participant identification, and AR on power-trend probes. The analysis relates these task differences to feature recoverability and backbone depth.

Critical strengths
  1. A well-motivated comparison with meaningful common controls. Sharing the tokenizer, backbone architecture, source data, and downstream evaluation removes several confounders that commonly arise when comparing separately developed foundation models. Nested participant-level validation for the consumer prediction tasks and the external TUAB/TUEV evaluations are useful. The VQ-only baseline also shows that the tokenizer itself already captures substantial EEG information.
Critical weaknesses
  1. The frozen tokenizer's receptive field is not specified clearly enough to establish that AR and MAE enforce their intended information restrictions. Section 2.1 says that a Transformer maps each patch to a feature, and Figure 1 places the tokenizer before the Stage-2 masking operations. It is unclear whether each feature h_i is computed independently from its raw patch or whether the tokenizer mixes information across the complete channel–time sequence. If the latter is bidirectional, an AR input feature could already contain future samples, and a visible MAE feature could already encode a subsequently masked patch. Freezing the tokenizer does not remove this dependency. This is a conditional concern, not a finding that leakage occurred. Please specify the attention layout and effective receptive field, ideally with the relevant implementation, and verify that perturbing raw future/hidden patches leaves the permitted input features unchanged. Patch-local encoding would resolve the ambiguity directly. If cross-patch context is used, the experiments should be described as prediction or masking of contextualized features, and the claims about future prediction and hidden-signal reconstruction reconsidered. Downstream labels need not be leaked for this issue to affect the objective interpretation.

  2. The experiment controls architecture more closely than it isolates the objective as the cause of each representation profile. The paper explicitly uses equal epochs rather than equal compute; MAE makes complementary masked passes, SLCL uses paired recordings and augmentations, and objective-specific optimization choices remain. These differences are legitimate components of the formulations, but the conclusions should remain conditional on the complete recipes and training budgets. Downstream folds or probe seeds also do not quantify pretraining-run variability. For AR in particular, a causal attention mask followed by mean pooling can itself create a different sensitivity to temporal position from a bidirectional encoder, so stronger power-trend probes do not uniquely establish a learned forecasting mechanism. These unresolved factors affect the central contribution: identifying how pretraining objectives shape representations. Supporting that contribution requires convergence/budget information, representative pretraining replication, a comparable-compute sensitivity analysis, and a control for the causal-mask/readout explanation. Restricting the interpretation to descriptive differences among the three recipes would reduce overclaiming, but would leave a narrower contribution than the objective-level explanation motivating the study. Precise configurations and an artifact-access plan are especially important because the principal corpus is private.

Questions for authors
  1. Is hih_i a function only of raw patch xix_i? What attention mask, if any, is used inside the Stage-1 encoder? Can a perturbation test verify that future or hidden raw patches do not alter features available to the Stage-2 predictor?

  2. How many independent pretraining runs were used for each objective and data fraction? Please provide actual training compute, convergence curves, chosen hyperparameters, and the data exposure of the shared tokenizer. Does participant exclusion from pretraining apply to both stages?

  3. Does AR's power-trend advantage remain with an appropriate causal-mask/readout control, and do the main rankings survive a representative replication or comparable-compute setting? These checks would determine how broadly the objective-specific conclusions can be stated.

7 comments
AuthorsOpenReview ↗

Response 1/3

Thank you for the careful review. Each control you requested is now in the revised manuscript: a patch-local tokenizer with all three objectives retrained, two additional pretraining seeds per objective, a compute-matched comparison, and untrained backbones with matching attention masks and readouts. With these controls, the objective-level conclusion holds: each objective performs best on the kind of task that its pretext task rewards, and the leading objective is unchanged on most tasks under each control. The untrained backbones further separate what is learned from what is architectural, and they show that what pretraining adds lies beyond classical EEG features and grows with depth. We answer each question in turn and then summarize what the controls establish.

Q1 — Tokenizer receptive field, attention masks, and perturbation test

1. Is each tokenizer feature a function of its raw patch alone?

No. In the original tokenizer the encoder attends bidirectionally across all 240 patches of a window, so each feature is contextualized by the full window. This follows the published NeuroLM and LaBraM tokenizer design, which we adopted so that each formulation is compared as it is used in practice. The revised Section 2.1 states the receptive field explicitly, and the Stage-2 tasks with this tokenizer are therefore prediction and masking over contextualized features.

2. What attention masks are used?

Stage 1 uses bidirectional attention across the window. At Stage 2, AR uses causal attention, while MAE and SLCL use bidirectional attention. These masks are integral to the formulations being compared, and untrained backbones with the same masks separate their architectural contribution (Q3).

3. Patch-local tokenizer and perturbation test.

Following your suggestion, we trained a patch-local tokenizer that normalizes each patch independently and restricts encoder attention to that patch. In the requested perturbation test, we kept one patch and replaced the other 239. Across five positions spanning the window, the maximum absolute change in the retained feature was 0.0, with no change in its assigned code. Reconstruction quality is unchanged (waveform/spectral losses of 0.668/0.314, versus 0.667/0.309 for the original tokenizer). With this tokenizer, AR predicts future codes strictly from past signal and MAE reconstructs codes strictly from visible signal.

We then retrained all three objectives with it (Appendix H, Table 9). The leading objective is unchanged on six of seven tasks, and on Hori AR and MAE differ by 0.005. The task-level conclusions therefore do not depend on cross-patch context in the tokenizer, and they hold when the tasks are future prediction and hidden-signal reconstruction in the strict sense. The layer-wise analyses in Figures 2 and 3 use the original tokenizer.

Q2 — Pretraining runs, compute, and data exposure

1. Independent pretraining runs.

The submitted paper used one pretraining run for each objective and data fraction. We added two seeds per objective at full data, giving three independent runs (Appendix I). Here ± is the standard deviation across the three pretraining runs.

TaskARMAESLCL
Sleep κ ↑0.667 ± 0.0010.660 ± 0.0270.624 ± 0.005
Hori κ ↑0.311 ± 0.0020.299 ± 0.0120.280 ± 0.002
Age MAE (years) ↓8.06 ± 0.017.95 ± 0.348.52 ± 0.02
Gender bal. acc. ↑0.678 ± 0.0020.709 ± 0.0140.641 ± 0.002
ISV top-1 ↑0.200 ± 0.0020.249 ± 0.0470.502 ± 0.013
TUAB bal. acc. ↑0.768 ± 0.0000.772 ± 0.0060.768 ± 0.003
TUEV bal. acc. ↑0.470 ± 0.0080.509 ± 0.0300.400 ± 0.002

The leading objective is unchanged on five of seven tasks. In every individual run, MAE leads gender and TUEV and SLCL leads ISV. On sleep and Hori, AR and MAE differ by no more than the variation across MAE runs.

2. Original training budget.

The original comparison used the same pretraining corpus and the same number of epochs (50) for each objective, which is a common basis for comparing self-supervised objectives. The cost per epoch differs because of how each objective is defined:

  • MAE masks 50% of the tokens and uses two passes with complementary masks, requiring two backbone passes per window.
  • AR computes all next-token predictions within the 60-second window in a single causal pass.
  • SLCL processes each window of a participant pair once and contrasts the pooled representations. | Objective | Epochs | H100 GPU-hours | |---|---:|---:| | AR | 50 | 59.0 | | MAE | 50 | 136.6 | | SLCL | 50 | 41.6 |
AuthorsOpenReview ↗

Response 2/3

Q2 — Pretraining runs, compute, and data exposure (continued)

3. Convergence curves and hyperparameters.

Only the peak learning rate was tuned across objectives, to obtain stable training: 4 × 10⁻⁴ for SLCL and 6 × 10⁻⁴ for AR and MAE. All other shared optimization settings are identical. Convergence curves are given in Appendices I and J, and the complete configurations in Appendix A.

HyperparameterStage 1: tokenizerStage 2: backbone
OptimizerAdamWAdamW
Adam β(0.9, 0.999)(0.9, 0.95)
Batch size4001,024
Weight decay10⁻⁴0.1
Peak learning rate2 × 10⁻⁵AR/MAE: 6 × 10⁻⁴; SLCL: 4 × 10⁻⁴
Minimum learning rate10⁻⁶6 × 10⁻⁵
ScheduleCosineCosine
Warmup epochs53

4. Compute-matched comparison.

We retrained AR and SLCL so that all three objectives use the same backbone compute as MAE, matched within 0.5% at approximately 1.01 × 10²⁰ FLOPs per run (Appendix J). Here ± is the standard deviation across evaluation folds or probe seeds.

TaskARMAESLCL
Sleep κ ↑0.657 ± 0.0170.692 ± 0.0080.636 ± 0.015
Hori κ ↑0.310 ± 0.0250.312 ± 0.0160.284 ± 0.022
Age MAE (years) ↓8.15 ± 0.227.61 ± 0.138.35 ± 0.16
Gender bal. acc. ↑0.674 ± 0.0130.722 ± 0.0080.679 ± 0.011
ISV top-1 ↑0.180 ± 0.0050.305 ± 0.0100.538 ± 0.003
TUAB bal. acc. ↑0.764 ± 0.0030.780 ± 0.0020.771 ± 0.006
TUEV bal. acc. ↑0.479 ± 0.0160.537 ± 0.0090.402 ± 0.025

The leading objective is unchanged on all seven tasks. Unequal backbone compute therefore does not explain the objective-specific strengths.

5. Tokenizer data exposure and participant exclusion.

Stage 1 and Stage 2 use the same pretraining corpus, and we verified that downstream evaluation participants are excluded from both stages. We will state this explicitly in the training protocol.

Q3 — Causal-mask and readout control, and robustness of the rankings

1. Does AR's power-trend advantage remain with a causal-mask and readout control?

We added the control you describe: an untrained backbone with the same architecture, causal attention, mean pooling, and probes as AR, receiving features from the same frozen tokenizer. To ensure that the control does not depend on one random draw of weights, we repeated it with three random initializations of the untrained backbone. The result separates two contributions.

  • Causal attention with mean pooling already makes power trends accessible. The untrained causal backbone reaches trend R² of 0.54–0.83, compared with 0.11–0.25 for the tokenizer alone. This confirms the mechanism you proposed.
  • AR pretraining adds a learned gain on top of it at almost every transformer block. AR exceeds all three untrained initializations at blocks 1–24 for the δ and β trends and at blocks 1–22 for the α trend. The gain is largest in the early blocks and decreases toward the output readout, where the two are comparable. The table reports AR minus untrained causal decoding R², averaged within four-layer groups. | Power trend | Layers 1–4 | 5–8 | 9–12 | 13–16 | 17–20 | 21–24 | |---|---:|---:|---:|---:|---:|---:| | δ | +0.119 | +0.095 | +0.080 | +0.070 | +0.054 | +0.028 | | α | +0.149 | +0.123 | +0.110 | +0.091 | +0.060 | +0.016 | | β | +0.110 | +0.102 | +0.098 | +0.089 | +0.074 | +0.043 |

The revised paper accordingly describes AR as a complete formulation, of which causal attention is an integral part (Sections 1 and 5), and we will add this trend-control table to Appendix F.

2. Is the forecasting advantage learned?

To test this downstream, we added sleep-onset trajectory prediction: from 60 seconds of EEG, predict the change in theta- and alpha-band power 1, 2, and 5 minutes later. Sleep onset provides a natural transition, as alpha activity subsides and theta activity becomes more prominent (Wright et al., 1995). Each model is compared with an untrained backbone that has the same attention mask and readout, which is the control you describe.

Part of AR's advantage is architectural and part is learned. The untrained causal backbone already scores higher than the bidirectional one (0.496 versus 0.475). On top of this, AR exceeds its untrained causal control by 0.028 R² at an intermediate block in each of three pretraining runs, and by 0.024 at matched compute, in every fold. MAE's gain over its control is about zero, and SLCL is below its control (Figure 2, Section 4.3).

3. Do the rankings survive replication and comparable compute?

Yes, as reported under Q2. Across three pretraining runs, the leading objective is unchanged on five of seven tasks, with AR and MAE within seed variation on the other two. At matched compute, it is unchanged on all seven.

AuthorsOpenReview ↗

Response 3/3

What the controls establish

Scope. Our conclusions concern three complete pretraining formulations, including their native attention masks and targets, under the reported training conditions. Untrained backbones with matching masks and readouts separate architectural contributions from learned ones.

Contribution. With these controls, the paper establishes more than descriptive differences among three recipes. The layer-wise analysis in Section 4.3, with Figures 2 and 3, is new in this revision and replaces the earlier feature-probing argument. We invite the reviewer to examine it.

  1. Each objective performs best on the kind of task that its pretext task rewards, and this holds under your controls. The leading objective is unchanged on most tasks with the patch-local tokenizer, across pretraining runs, and at matched compute (Q1–Q3, Appendices H–J).
  2. The gains are learned. On sleep, Hori, ISV, TUAB, and TUEV, untrained backbones remain within 0.016 of the tokenizer at every layer (Figure 2).
  3. Classical EEG features are a foundation that trained and untrained backbones share. We measure how much a score depends on six families of classical features by removing the information linearly associated with them from the representation (Section 2.3). On most tasks, this lowers the score of a trained backbone and of an untrained one by a similar amount (within 10% of each other in 16 of the 24 objective–task pairs at the final layer). Decoding these features therefore could not explain the downstream differences.
  4. What pretraining adds lies beyond these features, and it grows with depth. An untrained backbone depends on them for 84–86% of its sleep-staging score at every layer, whereas for MAE this share falls from 79% at layer 4 to 56% at layer 18 (Figure 3).
  5. A pretext task can also harm a representation, and the choice of layer changes how the objectives compare. SLCL retains subject specificity but falls below the untrained backbones on TUEV and sleep-onset trajectory prediction, which suggests that it may suppress changes within a participant. At intermediate depth, MAE's ISV accuracy approaches SLCL's (0.674 versus 0.708), but it falls to 0.463 at the output, so a final-layer comparison understates what MAE learns (Section 4.3). These findings support choosing the objective and the layer jointly for each task (Section 5). Revisions, point by point.
  • Tokenizer receptive field and attention layout (W1, Q1). Section 2.1 specifies that the original tokenizer attends across all patches of a window.
  • Patch-local encoding and perturbation test (W1, Q1). Appendix H reports the patch-local tokenizer, the locality test, and all three objectives retrained with it.
  • Pretraining replication (W2, Q2). Appendix I reports three pretraining runs per objective, with learning curves.
  • Training compute, convergence, and configurations (W2, Q2). Appendix A gives the complete configurations, and Appendices I and J the learning curves. We will add the GPU-hour table to Appendix A.
  • Comparable-compute sensitivity (W2, Q3). Appendix J reports the compute-matched comparison.
  • Tokenizer data exposure and participant exclusion (Q2). We will state in the training protocol that both stages use the same corpus and exclude the evaluation participants.
  • Causal-mask and readout control (W2, Q3). Figures 2 and 3 include untrained backbones with matching attention masks and readouts. We will add the layer-wise trend-control table to Appendix F.
  • Learned forecasting (W2, Q3). Sleep-onset trajectory prediction is added to Figure 2, with its definition in Appendix B and its final-layer seed and compute results in Appendices I and J. We will add it to Tables 2 and 3 in the camera-ready version.
  • Scope of the conclusions (W2). Sections 1 and 5 state that the comparison concerns complete pretraining formulations. Artifact access. We agree that precise configurations and access to artifacts are important when the principal corpus is private. The pretraining recordings are governed by a data-use agreement and cannot currently be released. The revised manuscript therefore documents the complete configurations (Appendix A), and three of the eight tasks (TUAB, TUEV, and sleep-onset trajectory prediction) use public datasets. We will pursue the permissions needed to share the code and pretrained models. In this paper, we provide the configurations and protocols required to reproduce the comparison on other corpora.

We appreciate these suggestions. They have strengthened the evidence for the comparison and clarified how the pretraining formulations shape the representations.

Reviewer_bhpdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how different self-supervised pretraining objectives influence EEG representations and downstream task performance. To reduce confounding factors introduced by different architectures and datasets, the authors systematically compare three objectives—masked autoencoding (MAE), autoregressive prediction (AR), and subject-level contrastive learning (SLCL)—using a shared frozen tokenizer and GPT-2 backbone. The experiments are conducted on 50,894 hours of EEG recordings and evaluated across seven downstream tasks, including sleep staging, age prediction, gender classification, subject identification, and clinical EEG analysis. The results show that MAE achieves the best performance on most downstream tasks, while SLCL excels at subject identification. Feature probing suggests that MAE better preserves spectral and cross-channel information, AR emphasizes temporal dynamics, and SLCL captures subject-specific characteristics. Additionally, intermediate representations generally outperform final-layer representations. These findings provide empirical insights into how pretraining objectives shape EEG representations and their suitability for different downstream tasks.

Critical strengths

1.The paper provides a systematic comparison of three representative EEG pretraining objectives using a shared tokenizer, backbone architecture, and data source. Evaluations across seven downstream tasks, different pretraining data scales, and LoRA fine-tuning settings offer useful empirical evidence about the relative strengths and limitations of these objectives. 2.Beyond downstream performance comparisons, the paper investigates how different objectives encode established EEG characteristics, including spectral information, cross-channel coherence, temporal power trends, and subject-specific features. The layer-wise analysis further reveals differences in representation transferability across model depth. These analyses provide valuable insights into the relationship between pretraining objectives and learned EEG representations.

Critical weaknesses

1.Although the paper aims to connect different pretraining objectives with their downstream strengths, the experiments provide limited evidence for systematic task-dependent objective selection. MAE performs best on most evaluated tasks, while SLCL's main advantage is concentrated on subject identification, which closely aligns with its pretraining objective. Meanwhile, AR demonstrates stronger temporal feature encoding without a corresponding advantage on downstream tasks. It therefore remains unclear under what task characteristics AR or SLCL should be preferred over MAE. 2.The feature-probing experiments reveal distinct spectral, spatial, and temporal characteristics across pretraining objectives. However, these observations primarily establish correlations rather than causal relationships between learned features and downstream performance. For example, it remains unclear whether MAE's stronger cross-channel coherence encoding directly contributes to its superior performance on specific tasks. Additional feature-level ablations or perturbation analyses would strengthen the proposed explanations. 3.Although the three objectives share the same backbone and training data, their computational costs and optimization procedures are not fully matched. Moreover, the shared tokenizer is trained using waveform and spectral reconstruction, potentially favoring objectives that preserve or exploit such information. Without compute-matched comparisons or analyses using alternative tokenizers, it is difficult to fully isolate the effects of the pretraining objectives themselves.

Questions for authors

1.The paper aims to connect different pretraining objectives with their downstream strengths. However, MAE performs best on most evaluated tasks, while SLCL's main advantage is limited to subject identification, which closely aligns with its pretraining objective. Meanwhile, AR demonstrates stronger temporal feature encoding without a corresponding advantage on downstream tasks. Could the authors clarify under what task characteristics or conditions AR or SLCL should be preferred over MAE? What evidence supports the practical selection of pretraining objectives based on downstream task requirements? 2.The feature-probing results reveal distinct spectral, spatial, and temporal characteristics across pretraining objectives. However, it remains unclear whether these representation differences directly explain the observed downstream performance. For example, does MAE's stronger cross-channel coherence encoding actually contribute to its superior performance on specific tasks? Could the authors provide feature-level ablations, perturbation experiments, or additional quantitative analyses to establish a stronger connection between the identified EEG characteristics and downstream performance? 3.Although the three pretraining objectives share the same tokenizer, backbone, and training data, their computational costs and optimization procedures are not fully matched. Moreover, the shared tokenizer is pretrained through waveform and spectral reconstruction, which may favor objectives relying on similar information. Could the authors clarify to what extent the observed differences are attributable to the pretraining objectives themselves rather than computational budget or tokenizer-specific biases? Would the main conclusions remain consistent under compute-matched training or alternative tokenization strategies?

3 comments
AuthorsOpenReview ↗

Response 1/2

Thank you for the constructive review. Your three questions shaped the revision. Each analysis you asked for is now in the manuscript, and we introduce each new experiment under the question that prompted it. The original conclusions hold under every new control.

Q1 — When should AR or SLCL be preferred over MAE?

1. Does AR's temporal encoding translate into a downstream advantage?

Yes. You noted that AR's temporal encoding had no corresponding downstream task, so we added one: sleep-onset trajectory prediction. From 60 seconds of EEG recorded while a person falls asleep, the model predicts how theta- and alpha-band power will change 1, 2, and 5 minutes later. AR predicts this trajectory best in each of three pretraining runs and at matched compute (final-layer scores in Appendices I and J). At an intermediate layer fixed in advance, its mean R² is 0.524, versus 0.476 for MAE and 0.394 for SLCL (additional analysis, reported here).

2. What is SLCL good for beyond the ISV benchmark?

SLCL keeps personal information even within a physiological state. To show this, we added a cross-night test that holds the sleep stage fixed and asks whether a model retrieves the same participant on a different night. SLCL's embeddings of the same person remain close together: it retrieves the participant roughly 1.5 to 2 times as often as MAE or AR, in every stage (Appendix K). SLCL therefore captures how one person's EEG differs from another's in the same state, which is the information needed to model individual differences and to adapt a model to a person.

3. Which objective should be preferred for which task?

Each objective performs best on the kind of task that its pretext task rewards (Section 5, Figure 2).

If the task requiresUseBecause its pretext task
The content of the signal (sleep stage, EEG events, age, gender)MAE, intermediate layersreconstructs masked content from its context
Subject-specific information (ISV)SLCL, output layeraligns recordings of the same participant
The future course of the signal (sleep-onset trajectory prediction)AR, intermediate layerspredicts the future from the past
  • MAE is the default. It transfers to the widest range of tasks, leading sleep staging, age prediction, gender classification, and EEG event classification. A plausible reason is that EEG is non-stationary: to fill in a masked segment from its context, the model must describe the signal as it is at each moment, and many tasks can use this description.
  • AR and SLCL have narrower strengths. AR is preferred when the task is temporal prediction, and SLCL when it requires subject-specific information.
  • A pretext task can also harm a representation. On EEG event classification and sleep-onset trajectory prediction, SLCL scores below the untrained backbones, which share the architecture and tokenizer but are never pretrained. Aligning recordings of the same participant may suppress the changes within a participant that these tasks draw on.

Q2 — Do the decoded EEG features contribute to downstream performance?

1. Can a feature-level ablation connect the features to downstream performance?

Yes. Following your suggestion, we added a feature-level ablation at every layer (new Figure 3). We group classical EEG features into six families, such as band power, coherence, and alpha-peak frequency. For each family, we remove from the representation everything that is linearly related to it, keep the linear model trained for the task unchanged, and measure how much the score falls. If a model's advantage came from the linear information about a feature family, removing it would remove the advantage (Section 2.3, Figure 3, Appendix G).

2. Does MAE's coherence encoding contribute to its superior performance?

Coherence contributes to the prediction, but it does not by itself account for MAE's advantage. It carries about a third of the lost score on age prediction and about a quarter on TUAB, a similar share for all three objectives. MAE retains an advantage over the untrained network after all the classical features are removed, so coherence alone is not what sets it apart.

AuthorsOpenReview ↗

Response 2/2

Q2 (continued)

3. What the ablation revealed.

The ablation you proposed did more than answer the question (Figure 3).

  • Classical features are a foundation that the objectives share. Each task relies on its own classical features, in line with known physiology: band power matters for every task and dominates sleep staging, age prediction also draws on alpha-peak frequency and coherence, and appreciable dependence on trends in power is concentrated on sleep-onset trajectory prediction. On most tasks, removing these features lowers the score of a pretrained network and of an untrained network by about the same amount. They therefore explain the task, not the objective.
  • What pretraining adds lies beyond these features, and it grows with depth. Take MAE on sleep staging as an example. In its early layers, about 80% of its performance depends on the classical features. At its best layer, about 55% does. In an untrained network, it is about 85% at every layer.
  • Each objective develops this gain at its own depth and for its own tasks. For SLCL it appears from the first layer on ISV, the task its pretext rewards, where at the output almost none of its accuracy depends on the classical features. For MAE it develops in the middle layers and is the largest of the three on age and gender, which no pretext task predicts. For AR it develops later and is smaller. On age prediction, pretraining also increases the use of the classical features themselves, for MAE and AR.

Q3 — Are the differences attributable to the objectives rather than compute or the tokenizer?

1. Would the conclusions hold at matched compute?

Yes. The original runs gave every objective the same number of epochs. MAE then uses more compute, because it makes two masked passes per window. Following your suggestion, we retrained AR and SLCL for longer until all three used the same backbone compute, matched within 0.5% (Appendix J). The leading objective is unchanged on all seven tasks.

2. Would they hold with an alternative tokenizer?

Yes. The original tokenizer encodes each one-second patch in the context of the whole window. We trained an alternative that encodes each patch independently and retrained all three objectives with it (Appendix H). The leading objective is unchanged on six of seven tasks, and on Hori AR and MAE differ by 0.005. The table gives both sets of results.

TaskMatched compute: AR / MAE / SLCLPatch-local tokenizer: AR / MAE / SLCL
Sleep κ ↑0.657 / 0.692 / 0.6360.697 / 0.710 / 0.645
Hori κ ↑0.310 / 0.312 / 0.2840.341 / 0.336 / 0.311
Age error (years) ↓8.15 / 7.61 / 8.357.73 / 7.12 / 8.53
Gender bal. acc. ↑0.674 / 0.722 / 0.6790.701 / 0.748 / 0.648
ISV top-1 ↑0.180 / 0.305 / 0.5380.378 / 0.438 / 0.552
TUAB bal. acc. ↑0.764 / 0.780 / 0.7710.786 / 0.794 / 0.772
TUEV bal. acc. ↑0.479 / 0.537 / 0.4020.504 / 0.524 / 0.429

3. Could a reconstruction-trained tokenizer favour particular objectives?

We think this is unlikely, for three reasons, although we cannot exclude it and the paper keeps it as a limitation (Section 5).

  • Reconstruction is a task-agnostic way to train a tokenizer. It asks the tokenizer to keep the signal itself, in time and in frequency, without reference to any label or downstream task.
  • All three objectives start from the same tokenizer features. Each backbone receives these features as its input, so whatever the tokenizer encodes is equally available to AR, MAE, and SLCL. An untrained backbone on these features scores about the same as the tokenizer alone, which gives the three objectives the same starting point.
  • What differs is what each objective does with these features. SLCL never predicts the codes, yet it reaches the highest ISV score, so the tokenizer does not prevent an objective from learning what its own task rewards.

What the revision establishes

The layer-wise analysis in Section 4.3, with Figures 2 and 3, is new and replaces the earlier feature-probing argument. We invite the reviewer to examine it.

Revisions, point by point.

  • New task, sleep-onset trajectory prediction (W1, Q1). Figure 2 and Appendices I and J. We will add it to Tables 2 and 3 in the camera-ready version.
  • New cross-night test of subject specificity (W1, Q1). Appendix K.
  • New untrained-backbone controls and layer-wise analysis (W1, Q1). Section 4.3 and Figure 2.
  • New feature-level ablation (W2, Q2). Section 2.3, Figure 3, and Appendix G.
  • New retraining at matched compute and with a patch-local tokenizer (W3, Q3). Appendices J and H. We appreciate these suggestions. They have turned a comparison of downstream scores into an account of what each objective learns and retains.

Dear Reviewer,

With two days remaining in the discussion period, we would appreciate your thoughts on our responses and additional experiments. If any concerns remain unresolved, please let us know so we can address them before the discussion closes.

Thank you for your time and consideration.

Reviewer_QqePOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper conducts empirical experiments to study several pretraining paradigms: masked autoencoding (MAE), autoregressive prediction (AR), and contrastive learning (SLCL). Beyond downstream performance analysis, this work probes the corresponding representations to gain clues about the encoded EEG features. Further layer-wise probing is conducted to further study the effect of the pretraining objective on evaluation.

Critical strengths
  1. Interesting idea of studying encoded features corresponding to the pretraining objective, despite clear motivation/application
  2. Extensive experiments to study and support the claims. The paper is well-structured
Critical weaknesses
  1. Limited novelty and technical contribution: The paper remains empirical and largely combines several existing works to analyze pretraining objectives further. Although stated in the contributions, the review paper [1] indirectly compares the performance of pretraining paradigms across several evaluations; LoRA fine-tuning is already established in [2]; and layer-wise analysis is well studied in previous work [3]. The most interesting point is empirical objective-specific feature encoding; however, this outcome does not clearly contribute.
  2. Unclear, insufficient explanations: The authors establish several features to test whether objective-specific pretraining can capture them, but do not explain why these features should be the case (rather than others), or whether these features are sufficient for an EEG foundation model. Moreover, despite stating that they study layer-wise performance for each objective, these peak observations do not seem systematic, and the authors do not interpret the results.

References

[1] EEG-FM-Bench: A Comprehensive Benchmark for the Systematic Evaluation and Diagnostic Analyses of EEG Foundation Models

[2] Are Large Brainwave Foundation Models Capable Yet ? Insights from Fine-Tuning, ICML 2025

[3] Mantis: Lightweight Foundation Model for Time Series Classification, ICML 2026

Questions for authors

Weaknesses above

3 comments
AuthorsOpenReview ↗

Response 1/2

Thank you for the thoughtful review and for pointing us to the three related works. Your questions about the choice of EEG features and the interpretation of the layer-wise results showed us that the connection between our representation analyses and the downstream findings was not clear enough. We therefore added feature-erasure experiments across depth (Figure 3), layer-wise comparisons with untrained backbones across eight tasks (Figure 2), and a new forecasting task, sleep-onset trajectory prediction.

Under shared data, tokenizer, and architecture, each objective performs best on the kind of task that its pretext task rewards: MAE transfers most broadly, SLCL is strongest on subject-specific information, and AR leads sleep-onset trajectory prediction. These strengths are built on classical EEG features that all three objectives share, and what pretraining adds lies beyond these features and grows with depth.

1. A systematic layer-wise analysis of the three objectives (W2)

1.1 What the models rely on: feature erasure (new Figure 3)

The feature-encoding analysis of the submitted version was a test of accessibility. It showed whether a feature can be decoded from a representation (now Appendix F). It did not show which downstream task uses that feature, so it could not connect the features to task performance.

The features are the common vocabulary of EEG analysis. We use six families: band power, spectral shape, coherence, alpha asymmetry, alpha-peak frequency, and trends in power. They are among the handcrafted features most widely used across EEG tasks, and each has a well-understood physiological meaning. They are not exhaustive, but they are the terms in which the field already describes EEG, so results stated in them can be interpreted.

Feature erasure tests whether a task actually uses a feature. At every layer, we remove from the representation everything that is linearly related to a feature family, keep the linear model trained for the task unchanged, and measure how much the task score falls. If the score falls, the task relied on that family; if it does not, the task did not. We do this for all three objectives, all eight tasks, and every layer, and also for untrained backbones, which are the same networks before any pretraining. Figure 3 taught us three things.

  • Each task depends on the classical features that the literature links to it (coloured bands in Figure 3). Band power (blue) contributes across tasks and dominates sleep staging, whereas age prediction also draws on alpha-peak frequency (red) and coherence (yellow). Appreciable dependence on power trends (dark green), meanwhile, is concentrated on sleep-onset trajectory prediction.
  • These features are a foundation that the three objectives share. On most tasks, the three objectives depend on them to about the same degree, and a network that was never pretrained depends on them just as much (similar colour proportions across columns, and black and grey arrows of similar length). Three tasks, however, depart from this pattern. On sleep-onset trajectory prediction, SLCL depends on power trends much less than AR and MAE (dark green band). On ISV, SLCL hardly depends on the features at the final layer (short black arrow), whereas AR and MAE continue to depend on them. On age, AR and MAE depend on them more than the untrained backbones (black arrows longer than grey).
  • What pretraining adds lies beyond these features, and it grows with depth (pink regions in Figure 3). In the early layers, most of the performance depends on the classical features: removing them lowers sleep-staging κ from about 0.6 to 0.1, and sleep-onset trajectory prediction R² from about 0.5 to below 0.1. In deeper layers, a pretrained network usually still scores higher than an untrained one after the features are removed. This difference, the gain beyond the features, is the part of the pretraining gain that the classical features do not explain, and it differs among the objectives.
    • ISV. SLCL's gain beyond the features appears from the first layer and eventually accounts for most of its performance. MAE's, in contrast, develops in the middle layers and partly recedes toward the output, while AR's remains small throughout.
    • Age and gender. Although no pretext task predicts these labels, MAE develops the largest gain beyond the features on both, from the middle layers onward. AR develops a smaller one later in the network. SLCL, however, shows only a small gain on age and none on gender: separating participants does not require the attributes they share with others. What we learned: classical features explain the task, not the objective, and the objectives differ largely in what pretraining adds beyond them. The classical features are therefore not sufficient to describe what a foundation model learns.
AuthorsOpenReview ↗

Response 2/2

1. A systematic layer-wise analysis of the three objectives (W2), continued

1.2 How each objective performs across layers (new Figure 2)

The objectives differ in how broadly they learn, how early their gains emerge, and how much they retain. To examine these differences, we added an evaluation of all eight tasks at every layer, using untrained backbones as references (Figure 2). Although 22 of the 24 curves peak at an intermediate layer and decline toward the output, the objectives follow distinct trajectories.

  • MAE transfers to the widest range of tasks, which suggests masked reconstruction as a default pretext task for EEG. Its gains develop by the middle layers. It is the best objective on six of the eight tasks, while its best-layer performance approaches the leader on the other two (ISV: 0.674 versus 0.708 for SLCL; sleep-onset trajectory prediction: 0.507 versus 0.533 for AR). A plausible reason is that EEG is non-stationary: to fill in a masked segment from its context, the model must describe the signal as it is at each moment, and many tasks can use this description. However, part of this advantage is lost toward the output, particularly on ISV, where SLCL's lead grows from 0.034 at their respective peaks to 0.200 at the final layer.
  • SLCL, in contrast, has a narrow strength, and its pretext task can also harm a representation. Its gains emerge within the first five layers, and it achieves the highest ISV performance. However, on EEG event classification and sleep-onset trajectory prediction, it remains below the untrained backbones from the first transformer layer onward. Together with its lower dependence on power trends, this suggests that aligning recordings of the same person may suppress the changes within a person that these two tasks draw on. This harm motivates comparing a pretrained model with an untrained network, which makes it visible, and combining subject-level contrast with an objective that preserves the content of the signal.
  • AR's strength is temporal prediction. Its gains emerge after MAE's on seven of the eight tasks. Nevertheless, it achieves the highest performance on sleep-onset trajectory prediction, where it exceeds its untrained backbone at intermediate layers, while remaining below MAE on the other tasks. What we learned: the choice of layer changes how the objectives compare, so the objective and the layer should be chosen together for each task.

2. New task: sleep-onset trajectory prediction (W1)

From 60 seconds of EEG recorded while a person falls asleep, the model predicts how theta- and alpha-band power will change 1, 2, and 5 minutes later. AR is the best objective on this task in each of three pretraining runs.

3. A recipe for choosing a pretext task (W1)

If the task requiresUse
The content of the signal (sleep stage, EEG events, age, gender)MAE, intermediate layers
Subject-specific information (ISV)SLCL, or MAE at an intermediate layer, which comes close
The future course of the signal (sleep-onset trajectory prediction)AR, intermediate layers

4. Relation to [1], [2], and [3] (W1)

These works compare and analyse models, whereas we compare pretraining methods. The models in [1] were developed independently, so they differ in pretraining data, tokenizer, and architecture as well as in method, and a difference between them cannot be traced to the method. In our study, the three models share the tokenizer, backbone architecture, corpus, and evaluation, and differ only in the pretraining method, including its native attention mask. Differences between them can therefore be traced to the method, up to run-to-run variation (Appendix I).

LoRA [2] and layer-wise analysis [3] are established tools, as you note, and we use them as such. With LoRA, we test whether the strengths of each pretraining method persist after adaptation, and they largely do (Section 4.1).

5. Revisions, point by point

  • New feature-erasure test at every layer (W2). Section 2.3, Figure 3, and Appendix G.
  • New layer-wise evaluation (W2). Section 4.3 and Figure 2.
  • New task, sleep-onset trajectory prediction (W1). Figure 2 and Appendix B; to be added to Tables 2 and 3 for camera ready.
  • Recipe and relation to prior work (W1). Sections 1 and 5. We appreciate these comments, and we invite the reviewer to examine Figures 2 and 3.

Dear Reviewer,

With two days remaining in the discussion period, we would appreciate your thoughts on our responses and additional experiments. If any concerns remain unresolved, please let us know so we can address them before the discussion closes.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.