- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper compares masked autoencoding, autoregressive prediction, and subject-level contrastive learning using a shared frozen EEG tokenizer and the same GPT-2 backbone architecture. Pretraining draws from a 50,894-hour private four-channel consumer-EEG corpus. Evaluation includes seven downstream tasks, LoRA adaptation, data-size comparisons, layer-wise probes, and probes of established spectral, spatial, and temporal EEG features. MAE reports the highest mean performance on six of seven downstream tasks, SLCL on cross-session participant identification, and AR on power-trend probes. The analysis relates these task differences to feature recoverability and backbone depth.
Critical strengths
- A well-motivated comparison with meaningful common controls. Sharing the tokenizer, backbone architecture, source data, and downstream evaluation removes several confounders that commonly arise when comparing separately developed foundation models. Nested participant-level validation for the consumer prediction tasks and the external TUAB/TUEV evaluations are useful. The VQ-only baseline also shows that the tokenizer itself already captures substantial EEG information.
Critical weaknesses
-
The frozen tokenizer's receptive field is not specified clearly enough to establish that AR and MAE enforce their intended information restrictions. Section 2.1 says that a Transformer maps each patch to a feature, and Figure 1 places the tokenizer before the Stage-2 masking operations. It is unclear whether each feature h_i is computed independently from its raw patch or whether the tokenizer mixes information across the complete channel–time sequence. If the latter is bidirectional, an AR input feature could already contain future samples, and a visible MAE feature could already encode a subsequently masked patch. Freezing the tokenizer does not remove this dependency. This is a conditional concern, not a finding that leakage occurred. Please specify the attention layout and effective receptive field, ideally with the relevant implementation, and verify that perturbing raw future/hidden patches leaves the permitted input features unchanged. Patch-local encoding would resolve the ambiguity directly. If cross-patch context is used, the experiments should be described as prediction or masking of contextualized features, and the claims about future prediction and hidden-signal reconstruction reconsidered. Downstream labels need not be leaked for this issue to affect the objective interpretation.
-
The experiment controls architecture more closely than it isolates the objective as the cause of each representation profile. The paper explicitly uses equal epochs rather than equal compute; MAE makes complementary masked passes, SLCL uses paired recordings and augmentations, and objective-specific optimization choices remain. These differences are legitimate components of the formulations, but the conclusions should remain conditional on the complete recipes and training budgets. Downstream folds or probe seeds also do not quantify pretraining-run variability. For AR in particular, a causal attention mask followed by mean pooling can itself create a different sensitivity to temporal position from a bidirectional encoder, so stronger power-trend probes do not uniquely establish a learned forecasting mechanism. These unresolved factors affect the central contribution: identifying how pretraining objectives shape representations. Supporting that contribution requires convergence/budget information, representative pretraining replication, a comparable-compute sensitivity analysis, and a control for the causal-mask/readout explanation. Restricting the interpretation to descriptive differences among the three recipes would reduce overclaiming, but would leave a narrower contribution than the objective-level explanation motivating the study. Precise configurations and an artifact-access plan are especially important because the principal corpus is private.
Questions for authors
-
Is a function only of raw patch ? What attention mask, if any, is used inside the Stage-1 encoder? Can a perturbation test verify that future or hidden raw patches do not alter features available to the Stage-2 predictor?
-
How many independent pretraining runs were used for each objective and data fraction? Please provide actual training compute, convergence curves, chosen hyperparameters, and the data exposure of the shared tokenizer. Does participant exclusion from pretraining apply to both stages?
-
Does AR's power-trend advantage remain with an appropriate causal-mask/readout control, and do the main rankings survive a representative replication or comparable-compute setting? These checks would determine how broadly the objective-specific conclusions can be stated.
7 comments
Response 2/3
Q2 — Pretraining runs, compute, and data exposure (continued)
3. Convergence curves and hyperparameters.
Only the peak learning rate was tuned across objectives, to obtain stable training: 4 × 10⁻⁴ for SLCL and 6 × 10⁻⁴ for AR and MAE. All other shared optimization settings are identical. Convergence curves are given in Appendices I and J, and the complete configurations in Appendix A.
| Hyperparameter | Stage 1: tokenizer | Stage 2: backbone |
|---|---|---|
| Optimizer | AdamW | AdamW |
| Adam β | (0.9, 0.999) | (0.9, 0.95) |
| Batch size | 400 | 1,024 |
| Weight decay | 10⁻⁴ | 0.1 |
| Peak learning rate | 2 × 10⁻⁵ | AR/MAE: 6 × 10⁻⁴; SLCL: 4 × 10⁻⁴ |
| Minimum learning rate | 10⁻⁶ | 6 × 10⁻⁵ |
| Schedule | Cosine | Cosine |
| Warmup epochs | 5 | 3 |
4. Compute-matched comparison.
We retrained AR and SLCL so that all three objectives use the same backbone compute as MAE, matched within 0.5% at approximately 1.01 × 10²⁰ FLOPs per run (Appendix J). Here ± is the standard deviation across evaluation folds or probe seeds.
| Task | AR | MAE | SLCL |
|---|---|---|---|
| Sleep κ ↑ | 0.657 ± 0.017 | 0.692 ± 0.008 | 0.636 ± 0.015 |
| Hori κ ↑ | 0.310 ± 0.025 | 0.312 ± 0.016 | 0.284 ± 0.022 |
| Age MAE (years) ↓ | 8.15 ± 0.22 | 7.61 ± 0.13 | 8.35 ± 0.16 |
| Gender bal. acc. ↑ | 0.674 ± 0.013 | 0.722 ± 0.008 | 0.679 ± 0.011 |
| ISV top-1 ↑ | 0.180 ± 0.005 | 0.305 ± 0.010 | 0.538 ± 0.003 |
| TUAB bal. acc. ↑ | 0.764 ± 0.003 | 0.780 ± 0.002 | 0.771 ± 0.006 |
| TUEV bal. acc. ↑ | 0.479 ± 0.016 | 0.537 ± 0.009 | 0.402 ± 0.025 |
The leading objective is unchanged on all seven tasks. Unequal backbone compute therefore does not explain the objective-specific strengths.
5. Tokenizer data exposure and participant exclusion.
Stage 1 and Stage 2 use the same pretraining corpus, and we verified that downstream evaluation participants are excluded from both stages. We will state this explicitly in the training protocol.
Q3 — Causal-mask and readout control, and robustness of the rankings
1. Does AR's power-trend advantage remain with a causal-mask and readout control?
We added the control you describe: an untrained backbone with the same architecture, causal attention, mean pooling, and probes as AR, receiving features from the same frozen tokenizer. To ensure that the control does not depend on one random draw of weights, we repeated it with three random initializations of the untrained backbone. The result separates two contributions.
- Causal attention with mean pooling already makes power trends accessible. The untrained causal backbone reaches trend R² of 0.54–0.83, compared with 0.11–0.25 for the tokenizer alone. This confirms the mechanism you proposed.
- AR pretraining adds a learned gain on top of it at almost every transformer block. AR exceeds all three untrained initializations at blocks 1–24 for the δ and β trends and at blocks 1–22 for the α trend. The gain is largest in the early blocks and decreases toward the output readout, where the two are comparable. The table reports AR minus untrained causal decoding R², averaged within four-layer groups. | Power trend | Layers 1–4 | 5–8 | 9–12 | 13–16 | 17–20 | 21–24 | |---|---:|---:|---:|---:|---:|---:| | δ | +0.119 | +0.095 | +0.080 | +0.070 | +0.054 | +0.028 | | α | +0.149 | +0.123 | +0.110 | +0.091 | +0.060 | +0.016 | | β | +0.110 | +0.102 | +0.098 | +0.089 | +0.074 | +0.043 |
The revised paper accordingly describes AR as a complete formulation, of which causal attention is an integral part (Sections 1 and 5), and we will add this trend-control table to Appendix F.
2. Is the forecasting advantage learned?
To test this downstream, we added sleep-onset trajectory prediction: from 60 seconds of EEG, predict the change in theta- and alpha-band power 1, 2, and 5 minutes later. Sleep onset provides a natural transition, as alpha activity subsides and theta activity becomes more prominent (Wright et al., 1995). Each model is compared with an untrained backbone that has the same attention mask and readout, which is the control you describe.
Part of AR's advantage is architectural and part is learned. The untrained causal backbone already scores higher than the bidirectional one (0.496 versus 0.475). On top of this, AR exceeds its untrained causal control by 0.028 R² at an intermediate block in each of three pretraining runs, and by 0.024 at matched compute, in every fold. MAE's gain over its control is about zero, and SLCL is below its control (Figure 2, Section 4.3).
3. Do the rankings survive replication and comparable compute?
Yes, as reported under Q2. Across three pretraining runs, the leading objective is unchanged on five of seven tasks, with AR and MAE within seed variation on the other two. At matched compute, it is unchanged on all seven.
Response 3/3
What the controls establish
Scope. Our conclusions concern three complete pretraining formulations, including their native attention masks and targets, under the reported training conditions. Untrained backbones with matching masks and readouts separate architectural contributions from learned ones.
Contribution. With these controls, the paper establishes more than descriptive differences among three recipes. The layer-wise analysis in Section 4.3, with Figures 2 and 3, is new in this revision and replaces the earlier feature-probing argument. We invite the reviewer to examine it.
- Each objective performs best on the kind of task that its pretext task rewards, and this holds under your controls. The leading objective is unchanged on most tasks with the patch-local tokenizer, across pretraining runs, and at matched compute (Q1–Q3, Appendices H–J).
- The gains are learned. On sleep, Hori, ISV, TUAB, and TUEV, untrained backbones remain within 0.016 of the tokenizer at every layer (Figure 2).
- Classical EEG features are a foundation that trained and untrained backbones share. We measure how much a score depends on six families of classical features by removing the information linearly associated with them from the representation (Section 2.3). On most tasks, this lowers the score of a trained backbone and of an untrained one by a similar amount (within 10% of each other in 16 of the 24 objective–task pairs at the final layer). Decoding these features therefore could not explain the downstream differences.
- What pretraining adds lies beyond these features, and it grows with depth. An untrained backbone depends on them for 84–86% of its sleep-staging score at every layer, whereas for MAE this share falls from 79% at layer 4 to 56% at layer 18 (Figure 3).
- A pretext task can also harm a representation, and the choice of layer changes how the objectives compare. SLCL retains subject specificity but falls below the untrained backbones on TUEV and sleep-onset trajectory prediction, which suggests that it may suppress changes within a participant. At intermediate depth, MAE's ISV accuracy approaches SLCL's (0.674 versus 0.708), but it falls to 0.463 at the output, so a final-layer comparison understates what MAE learns (Section 4.3). These findings support choosing the objective and the layer jointly for each task (Section 5). Revisions, point by point.
- Tokenizer receptive field and attention layout (W1, Q1). Section 2.1 specifies that the original tokenizer attends across all patches of a window.
- Patch-local encoding and perturbation test (W1, Q1). Appendix H reports the patch-local tokenizer, the locality test, and all three objectives retrained with it.
- Pretraining replication (W2, Q2). Appendix I reports three pretraining runs per objective, with learning curves.
- Training compute, convergence, and configurations (W2, Q2). Appendix A gives the complete configurations, and Appendices I and J the learning curves. We will add the GPU-hour table to Appendix A.
- Comparable-compute sensitivity (W2, Q3). Appendix J reports the compute-matched comparison.
- Tokenizer data exposure and participant exclusion (Q2). We will state in the training protocol that both stages use the same corpus and exclude the evaluation participants.
- Causal-mask and readout control (W2, Q3). Figures 2 and 3 include untrained backbones with matching attention masks and readouts. We will add the layer-wise trend-control table to Appendix F.
- Learned forecasting (W2, Q3). Sleep-onset trajectory prediction is added to Figure 2, with its definition in Appendix B and its final-layer seed and compute results in Appendices I and J. We will add it to Tables 2 and 3 in the camera-ready version.
- Scope of the conclusions (W2). Sections 1 and 5 state that the comparison concerns complete pretraining formulations. Artifact access. We agree that precise configurations and access to artifacts are important when the principal corpus is private. The pretraining recordings are governed by a data-use agreement and cannot currently be released. The revised manuscript therefore documents the complete configurations (Appendix A), and three of the eight tasks (TUAB, TUEV, and sleep-onset trajectory prediction) use public datasets. We will pursue the permissions needed to share the code and pretrained models. In this paper, we provide the configurations and protocols required to reproduce the comparison on other corpora.
We appreciate these suggestions. They have strengthened the evidence for the comparison and clarified how the pretraining formulations shape the representations.
Response 1/3
Thank you for the careful review. Each control you requested is now in the revised manuscript: a patch-local tokenizer with all three objectives retrained, two additional pretraining seeds per objective, a compute-matched comparison, and untrained backbones with matching attention masks and readouts. With these controls, the objective-level conclusion holds: each objective performs best on the kind of task that its pretext task rewards, and the leading objective is unchanged on most tasks under each control. The untrained backbones further separate what is learned from what is architectural, and they show that what pretraining adds lies beyond classical EEG features and grows with depth. We answer each question in turn and then summarize what the controls establish.
Q1 — Tokenizer receptive field, attention masks, and perturbation test
1. Is each tokenizer feature a function of its raw patch alone?
No. In the original tokenizer the encoder attends bidirectionally across all 240 patches of a window, so each feature is contextualized by the full window. This follows the published NeuroLM and LaBraM tokenizer design, which we adopted so that each formulation is compared as it is used in practice. The revised Section 2.1 states the receptive field explicitly, and the Stage-2 tasks with this tokenizer are therefore prediction and masking over contextualized features.
2. What attention masks are used?
Stage 1 uses bidirectional attention across the window. At Stage 2, AR uses causal attention, while MAE and SLCL use bidirectional attention. These masks are integral to the formulations being compared, and untrained backbones with the same masks separate their architectural contribution (Q3).
3. Patch-local tokenizer and perturbation test.
Following your suggestion, we trained a patch-local tokenizer that normalizes each patch independently and restricts encoder attention to that patch. In the requested perturbation test, we kept one patch and replaced the other 239. Across five positions spanning the window, the maximum absolute change in the retained feature was 0.0, with no change in its assigned code. Reconstruction quality is unchanged (waveform/spectral losses of 0.668/0.314, versus 0.667/0.309 for the original tokenizer). With this tokenizer, AR predicts future codes strictly from past signal and MAE reconstructs codes strictly from visible signal.
We then retrained all three objectives with it (Appendix H, Table 9). The leading objective is unchanged on six of seven tasks, and on Hori AR and MAE differ by 0.005. The task-level conclusions therefore do not depend on cross-patch context in the tokenizer, and they hold when the tasks are future prediction and hidden-signal reconstruction in the strict sense. The layer-wise analyses in Figures 2 and 3 use the original tokenizer.
Q2 — Pretraining runs, compute, and data exposure
1. Independent pretraining runs.
The submitted paper used one pretraining run for each objective and data fraction. We added two seeds per objective at full data, giving three independent runs (Appendix I). Here ± is the standard deviation across the three pretraining runs.
The leading objective is unchanged on five of seven tasks. In every individual run, MAE leads gender and TUEV and SLCL leads ISV. On sleep and Hori, AR and MAE differ by no more than the variation across MAE runs.
2. Original training budget.
The original comparison used the same pretraining corpus and the same number of epochs (50) for each objective, which is a common basis for comparing self-supervised objectives. The cost per epoch differs because of how each objective is defined: