acceptodds
Under review as a conference paper at ICLR 2027

Reconstruction Wastes Rate: Selective Prediction over Code Targets for Biosignal Foundation Models

Abstract

Discrete tokenizers for physiological signals are typically trained to reconstruct waveforms or spectral features. Reconstruction rewards encoding any component that lowers squared error, including noise that the surrounding context cannot predict, whereas a context model benefits from codes that the context can predict. We introduce (Selective Prediction Over Targets), which trains a context encoder to identify the quantized teacher code of a masked signal cell among candidates, without reconstructing the waveform or spectrum. On clinical EEG with injected noise, a probe recovers more of the noise from representations learned with reconstruction than from those learned with SPOT. We use SPOT to train CORTEX, a channel-agnostic encoder for EEG, EOG, EMG, ECG, and respiratory channels, and compare it with eleven baselines on eight clinical tasks spanning sleep and epilepsy. We show that is robust to increasing input noise and largely retains its clean-data performance under muscle-band contamination. A controlled comparison on an existing backbone, varying only the pretraining objective, attributes this robustness to the objective rather than the architecture. These results suggest that discriminative, prediction-based objectives yield representations that are more robust to noise than reconstructive ones, with direct implications for physiological signal encoders deployed in real-world, low-SNR clinical settings.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG foundation models, self-supervised learning, representation learning for biosignals, predictive coding, discrete tokenization, vector quantization, masked prediction, information bottleneck, robustness to noise, sleep staging, seizure detection, channel-agnostic architectures

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Lu6TOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work introduces SPOT (Selective Prediction Over Targets) a pretraining objective for EEG models that does not rely on signal reconstruction but rather signal selection. It also introduces CORTEX that uses SPOT to train a biosignal tokenizer. The authors have conducted a series of experiments with clean and noisy data to test their hypotheses.

Critical strengths

The paper is very well-written. The proposed methods are oriented around the hypothesis that reconstruction models use much of their capacity to reconstruct noise present at EEG signals. The main reason behind it is that MSE reconstruction allocates capacity over the directions with the most variance, regardless of whether this variance comes from signal or noise. They pretrain CORTEX using SPOT (the two proposed methodologies). I like the noise analysis with the synthetic data. The SPOT methodology is also model agnostic which makes it efficient for many other EEG FM architectures.

Critical weaknesses

However, I find the evidence not convincing enough. First of all, the analysis on synthetic data does not follow on real data (as discussed by the authors) or it is very over-simplified. Regarding the novelty, I do not see how SPOT is effectively different than any other codebook-based model (e.g LaBraM, NeuroRVQ). The objective is different that is true (InfoNCE instead of cross-entropy) but these models do not actually do reconstruction. They predict the closest codebook (in the latent space) for the masked token during pre-training. So I understand the authors' claim about that a reconstruction tokenizer (stage 1 at LaBraM) learns also noise but both approaches learn a specific set of latent vectors and it's the model's pre-training job to decide which ones to use, essentially choosing from nearby content. To that argument the authors have an ablation with LaBraM but I don't see the effectiveness (see questions below): LaBraM-recon is better than LaBraM-SPOT. I also have concerns about the results on Table 1. Cortex's pre-training corpus task distribution matches the downstream tasks while other EEG FMs have a broader pre-training corpus. This alignment can inflate performance. In addition, the authors only did probing which if not done properly can severely affect results - e.g. discussed in "Are EEG Foundation Models Worth It? Comparative Evaluation with Traditional Decoders in Diverse BCI Tasks" (ICLR2026). There is a lack of full-fine tuning results and benchmarks outside the pre-training corpus distribution.

Questions for authors
  1. How were the LaBraM-recon and LaBraM-spot trained ? Did the authors use the pre-trained models ? Did they re-train both tokernizer and LaBraM model from scratch using both settings ? How do the authors justify the results ?
  2. Could the authors perform a comparison where some FMs are pre-trained on the exact same corpus as CORTEX to alleviate distribution match concerns ? And evaluation on tasks outside clinical (sleep and seizure) ?
  3. About novelty, how is the method fundamentally/theoretically different besides the technical implementation than codebook-based EEG FMs ? The authors' claim about capturing noise applies only to the tokenisers.
  4. How do the authors create the positive and negative pairs ? If they are random maybe the model can use shortcuts like subject-id etc.
  5. How about 4096 or the InfoNCE temperature choices ? Any ablations on those parameters ?
5 comments
AuthorsOpenReview ↗

Questions: novelty.

Reply to Reviewer 3 (1/5). Part A, your questions. Question: how is SPOT different from codebook models such as LaBraM and NeuroRVQ?

Thank you for the detailed review and for the pointer to the comparison with traditional decoders. We answer your questions first (replies 1/5 to 3/5), and then the weaknesses (replies 4/5 and 5/5). We start with novelty because several other points depend on it. This comment also answers the novelty point in your weaknesses.

The difference is in how the code targets are built, not in the loss. We agree that stage 2 of LaBraM and NeuroRVQ predicts codes and does not reconstruct. But those codes come from a tokenizer that was first trained to reconstruct the signal, and stage 2 can only choose among them. In SPOT the targets come from an EMA teacher of the context encoder, trained only through the prediction loss, with no reconstruction term anywhere. So our claim is about where the codes come from, and this is also why it applies to codebook models even though their stage 2 does not reconstruct.

The theoretical difference. Theorem 1 is about exactly this step. A reconstruction-trained tokenizer allocates its capacity by input variance (part ii), so when noise dominates some directions, part of every code describes that noise. Stage 2 must then learn to tell apart codes that differ only in noise the context cannot predict, which is impossible to do from context and wastes the target vocabulary. A predictive code allocates capacity by what the context predicts (part i). SPOT moves the predictive principle from stage 2 into the tokenizer, so the codes are shaped by predictability from the start.

A test that separates the two. On the CORTEX backbone we crossed (1) the source of the target codes, a finished reconstruction model ("recon codes", as in LaBraM-style models) or a finished SPOT model ("SPOT codes"), with (2) the loss used to predict them, cross-entropy (as in LaBraM-style models) or InfoNCE (as in SPOT). The row "recon codes, cross-entropy" is therefore the LaBraM and NeuroRVQ recipe on our backbone. In every run the student is a fresh CORTEX with the main SPOT recipe, the teacher is frozen, and both kinds of code go through the same fixed quantizer. Each combination was trained twice (two student seeds). Scores are under 12 dB EMG, mean of the two runs.

Task (metric)LossRecon codesSPOT codesGain
TUH seizure (AUPRC)cross-entropy56.770.3+13.6
HMC staging (kappa)cross-entropy39.853.2+13.3
Arousal (AUPRC)cross-entropy50.153.5+3.5
TUH seizure (AUPRC)InfoNCE63.770.2+6.5
HMC staging (kappa)InfoNCE47.152.9+5.8
Arousal (AUPRC)InfoNCE49.153.7+4.5

SPOT codes score higher in all six comparisons, and in each run separately (smallest single-run gain +1.8). Replacing recon codes by SPOT codes raises the score by 8 points on average (3.5 to 13.6). Changing the loss with the codes fixed matters less, about 3 points on average, and with SPOT codes 0.3 points or less. So most of the difference comes from the codes, not the loss, which is the novelty we claim. One limit: our recon codes come from a reconstruction-trained encoder and a fixed quantizer, so this tests the idea and not LaBraM's exact learned codebook. Both runs are in the appendix section "Target codes versus loss".

AuthorsOpenReview ↗

LaBraM training and same corpus.

Reply to Reviewer 3 (2/5). Part A, your questions. Questions: how were LaBraM-recon and LaBraM-SPOT trained, and can some models be pretrained on the same corpus as CORTEX?

How the two LaBraM arms were trained. Neither arm uses a pretrained LaBraM checkpoint. Both were trained from scratch on our pooled corpus with the same settings: batch size 64, 100 epochs, AdamW, the same data and sampling. Their stored training arguments differ only in the model name and the output paths. Both use LaBraM's encoder (5,818,976 parameters, identical state_dict keys and shapes). LaBraM-recon is LaBraM's stage-1 VQ-NSP tokenizer (reconstructs Fourier amplitude and phase through an 8192-entry codebook and a decoder; no masking). LaBraM-SPOT is the same encoder trained with SPOT (75% masking, EMA teacher with decay 0.999, residual FSQ with levels 8^8 and depth 5, InfoNCE with up to 4096 candidates; no decoder, no reconstruction term). Neither arm trains a stage 2; both are probed directly from the final-epoch checkpoint with the same frozen probe. Masking, a teacher and a quantizer come with predicting a code from context, so this comparison is supporting evidence; the single-factor test is the 2x2 on CORTEX (reply 1/5). Details: appendix section "The LaBraM arms".

How we justify the results. On clean signals the results depend on the task (LaBraM-recon higher on HSP staging, HMC and arousal; LaBraM-SPOT higher on TUH and Siena), and we do not claim an advantage on clean signals. Under 12 dB EMG, LaBraM-SPOT is higher on every task above the trivial floor: HSP staging 33.9 against 27.8, HMC 27.4 against 0.4, arousal 22.6 against 17.1, TUH 55.8 against 22.4, Siena 27.3 against 2.8. Our claim is about robustness, and it holds on both backbones.

Pretraining on the same corpus. We agree that this is the key control. Three of our comparisons hold the corpus fixed exactly, so corpus alignment cannot explain them:

  1. CORTEX-recon against CORTEX-SPOT, three pretraining seeds each: SPOT is ahead under 12 dB EMG in 9 of 9 task-by-seed comparisons (+8.7 +- 5.0) and equal on clean signals (-1.0 +- 1.6).
  2. LaBraM-recon against LaBraM-SPOT (above).
  3. The 2x2 test of target codes and loss (reply 1/5).

Corpora never seen in pretraining. Four of our eight evaluations use corpora that are not in our pretraining data. CORTEX-SPOT has the best frozen score at 12 dB on all four, by a clear margin over the next frozen model:

Corpus (metric)CORTEX-SPOTNext best frozen model
Sleep-EDF (kappa)67.258.9 (NeuroRVQ)
HMC (kappa)51.743.7 (NeuroRVQ)
CHB-MIT (AUPRC)46.624.5 (BIOT)
Siena (AUPRC)30.29.2 (REVE)

On clean signals it is best on two of the four (Sleep-EDF and Siena). The overlap also runs in both directions: LaBraM and CBraMod were themselves pretrained on data that include the TUH EEG corpus. Pretraining all six published models on our corpus is not possible within the discussion period; the Limitations section now states that they are evaluated from their released checkpoints.

AuthorsOpenReview ↗

Questions: positives and negatives, number of negatives and temperature.

Reply to Reviewer 3 (3/5). Part A, your questions. Questions: how are the positive and negative pairs created, can the model use shortcuts such as the subject, and what about 4096 negatives and the temperature?

Positives and negatives. The positive for each masked cell is the EMA teacher's code for that cell, computed from the unmasked recording. The negatives are the teacher codes of 4096 other masked cells in the batch, drawn without regard to subject, so they can come from the same recording (Section 2.1).

Shortcuts such as subject identity. A subject-identity shortcut could help against negatives from other subjects, but it cannot help against negatives from the same recording. We therefore trained CORTEX-SPOT with negatives restricted to the same recording, and with negatives restricted to other recordings (one pretraining run and one probe seed each).

ModelTUH, clean signalsTUH, 12 dBHMC, clean signalsHMC, 12 dBArousal, clean signalsArousal, 12 dB
All other masked cells (main)71.170.452.851.657.654.4
Same recording only69.168.849.847.356.454.8
Other recordings only69.468.753.451.856.052.8

Performance on clean signals stays within 3 points of the main setting in either case. We also tried to decode identity from the frozen embeddings. Session identity (114 classes) is decoded with 65.2% accuracy from SPOT and 70.4% from reconstruction, and patient identity (42 classes) with 85.0% and 87.1%. Both encoders carry identity information, and SPOT carries slightly less. We find no evidence that SPOT relies on a subject-identity shortcut. This is in the appendix section "Negative source and identity probes".

4096 negatives and the temperature. Yes, we ran both ablations (table in the shared-results comment, and in the appendix section "Ablations of the SPOT objective").

ModelTUH, clean signalsTUH, 12 dBHMC, clean signalsHMC, 12 dB
256 negatives63.363.546.444.7
1024 negatives66.958.950.048.7
4096 (main), learned temperature71.170.452.851.6
Temperature 0.0268.469.052.551.1
Temperature 0.271.569.951.548.8

More negatives improve performance on clean signals, and the result changes little over the two temperatures. For reference, CORTEX-recon scores 59.4 on TUH and 47.7 on HMC at 12 dB. SPOT is ahead of it at 4096 negatives and at both temperatures. With fewer negatives it is ahead on one of the two tasks and behind on the other. The temperature of the main setting is now stated in the pretraining table of the appendix.

Reviewer_fMLhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the author introduced SPOT, a predictive objective for learning discrete biosignal targets, and evaluated it with the CORTEX encoder. The author shows that contextual prediction can improve robustness to noise compared with reconstruction. The author validates this hypothesis through evaluations across eight sleep and seizure tasks.

Critical strengths

First, I think the author highlights an important problem, where reconstruction objectives put more focus on capturing variations from the input, while downstream models additionally need context information. Moreover, the author shows that, when comparing reconstruction and SPOT on both CORTEX and LaBraM, SPOT improves performance under contamination in both backbones. This shows the generalizability of CORTEX across different pre-trained architectures. Moreover, I think the authors did some intensive work experimenting with eight sleep and seizure datasets and compared them with many baselines.

Critical weaknesses

I found the work to have the following weaknesses:

  1. The results section reads as if it were written by strong AI elements, it makes the reading very difficult. For example, a title like clean performance does not make any sense, and I would highly suspect it comes from AI. I think in general, people would just say performance on clean signals.

  2. The theorem indicates an ideal predictive code under a Gaussian model. However, SPOT is trained differently, where it predicts FSQ codes from an EMA teacher using InfoNCE, with negatives drawn from other samples that may come from the same recording or subject. So I think there is some mismatch between how it is trained and what the theorem might specify.

  3. The paper does not provide ablation studies on many factors such as the number of negative candidates, FSQ depth, EMA decay, or tube masking. As a result, it is difficult to tell whether the performance improvement comes from the predictive objective itself or from particular parameters in its implementation. I don't think simple acknowledgment of this is enough in the limitations section.

Questions for authors

I found the work to have the following weaknesses:

  1. The results section reads as if it were written by strong AI elements, it makes the reading very difficult. For example, a title like clean performance does not make any sense, and I would highly suspect it comes from AI. I think in general, people would just say performance on clean signals.

  2. The theorem indicates an ideal predictive code under a Gaussian model. However, SPOT is trained differently, where it predicts FSQ codes from an EMA teacher using InfoNCE, with negatives drawn from other samples that may come from the same recording or subject. So I think there is some mismatch between how it is trained and what the theorem might specify.

  3. The paper does not provide ablation studies on many factors such as the number of negative candidates, FSQ depth, EMA decay, or tube masking. As a result, it is difficult to tell whether the performance improvement comes from the predictive objective itself or from particular parameters in its implementation. I don't think simple acknowledgment of this is enough in the limitations section.

3 comments
AuthorsOpenReview ↗

R2.2 theorem versus actual training.

Reply to Reviewer 2 (1/2): the theorem and the trained system

Thank you for the review. We start with your technical point.

R2.2 Mismatch between the theorem and how SPOT is trained. We agree, and the paper now states the scope of the theorem more precisely. The theorem compares the optima of the two objectives under a Gaussian model. It does not describe the trained system, and it does not guarantee that training reaches the optimum. We use it for one purpose: it tells us which quantity to measure (the share of a code spent on directions that the context cannot predict) and in which direction the two objectives should differ. The trained system departs from the model in three ways. Section 3 now lists them, and we tested each one.

The theorem assumesSPOT usesHow we tested itResult
A target that depends only on what the context can predictTarget codes from an EMA teacher that sees the cell with its noiseSynthetic model with EMA-teacher targets, 5 seedsMedian noise share 3.0%, 1.6% and 0.4% at -10, -6 and -3 dB, against 99.8% for reconstruction at -10 dB
A continuous Gaussian code under a rate budgetResidual FSQ codesFSQ depth 1, 3 and 5 on real EEGSPOT ahead of reconstruction at 12 dB in 9 of 9 depth-by-task comparisons
Mutual information between code and contextInfoNCE with in-batch negatives that can come from the same recording or subjectNegatives only from the same recording, or only from other recordings, plus identity probesClean scores within 3 points of the main setting; ahead of reconstruction at 12 dB in 5 of 6 comparisons; identity decoded slightly less from SPOT

Why the mechanism carries over (intuition, not a proof). The teacher in SPOT is an EMA copy of the context encoder, and the context encoder is rewarded only for what it can predict about a hidden cell from the cells around it. Noise in the hidden cell that the context cannot predict gives the encoder nothing to gain, so the features that the teacher inherits are mostly the ones the context predicts. The teacher can still pass on some noise, because it sees the noisy cell. This is the drift we measure in the synthetic variant: a median noise share of at most 3.0%, and 7.5% for the largest single seed (seed 15 at -6 dB). A reconstruction objective, in contrast, is rewarded directly for encoding that noise.

Negatives from the same recording or subject. Because of these negatives, InfoNCE is not a certified bound on mutual information, and the paper treats it as a training surrogate (appendix section "InfoNCE as a contrastive surrogate"). Restricting the negatives does not change the conclusion:

Negatives fromTUH clean / 12 dBHMC clean / 12 dBArousal clean / 12 dB
All other masked cells (main)71.1 / 70.452.8 / 51.657.6 / 54.4
Same recording only69.1 / 68.849.8 / 47.356.4 / 54.8
Other recordings only69.4 / 68.753.4 / 51.856.0 / 52.8
CORTEX-recon (3 seeds)70.7 / 59.454.6 / 47.758.8 / 41.6

The only comparison where SPOT is not ahead at 12 dB is HMC with same-recording negatives (47.3 against 47.7).

What changed in the paper. Remark 1 in Section 3 now says that the theorem is a statement about optima that tells us what to measure, and lists the three departures above with pointers to the appendix sections "Synthetic variant with the training machinery", "Ablations of the SPOT objective" and "Negative source and identity probes".

AuthorsOpenReview ↗

R2.1 writing and R2.3 ablations.

Reply to Reviewer 2 (2/2): writing and ablations

R2.1 Writing. Thank you for the direct feedback, and we agree. We rewrote Sections 4.1 to 4.5 in plainer wording, so that each heading and sentence says what is measured. We took your suggestion for the heading: "Clean performance" is now "Performance on clean signals". We wrote the new results in the same plain way.

R2.3 Ablations. We agree that acknowledging this in the limitations is not enough, so we ran the ablations and added them to the paper (appendix section "Ablations of the SPOT objective", with a pointer from Section 4.1). We trained one-factor-at-a-time variants of CORTEX-SPOT: number of negatives, temperature, FSQ depth, EMA decay and masking. Each is a full 70-epoch pretraining with the other settings at the main configuration. We tested each on TUH seizure detection, HMC sleep staging and HSP arousal detection, on clean signals and under 12 dB and 0 dB EMG. CORTEX-recon (mean of three pretraining seeds) is the reference. The full table is in our shared-results comment.

Your question was whether the improvement comes from the predictive objective itself or from particular parameters. The results point to the objective:

  • The main setting is best or tied in performance on clean TUH signals in every row.
  • The SPOT variants are ahead of reconstruction at 12 dB in 32 of 36 comparisons and at 0 dB in 25 of 36. So the improvement does not depend on one value of one setting.
  • With random-cell masking instead of our tube masking, SPOT is still ahead of reconstruction at 12 dB on all three tasks (TUH 63.8 against 59.4, HMC 49.5 against 47.7, arousal 52.3 against 41.6), and at 0 dB on HMC and arousal. The variance-weighted tube raises performance on clean TUH signals by about 5 points over random-cell masking, but the 12 dB advantage does not depend on it.
  • The EMA decay matters most: 0.99 and 0.999 are 9.7 and 7.2 points below the main setting (0.9999) in performance on clean TUH signals. This is consistent with stable teacher targets mattering.

Dear authors,

I have read the rebuttal and checked the updated text. However, I remain my feeling on, for example, writing is not very human nature, and the added ablation reads difficult.

I remain my score.

Thanks

Reviewer_8Hq1OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that existing EEG FMs face a bottleneck, which is that reconstruction-based objectives may preserve noise rather than useful information for downstream tasks. To address this problem, the authors propose SPOT, a selective prediction objective that predicts discrete teacher codes from surrounding context instead of reconstructing the original signals. The authors also propose CORTEX, a channel-agnostic encoder for different biosignals. The model is pretrained on TUH and HSP recordings and evaluated on 8 clinical tasks to show the improvements over previous EEG foundation models. The authors also investigate how reconstruction and prediction objectives allocate representation capacity under noise.

Critical strengths
  1. This paper addresses a very relevant research gap, especially the problem that reconstruction objectives may spend representation capacity on noise rather than useful physiological information.

  2. The authors are careful with the experimental design, using 8 clinical evaluations and comparisons with 11 baselines. I especially appreciate the controlled experiments in Sec. 4.5, where the authors apply the same objective to both CORTEX and LaBraM backbones. I think these experiments provide convincing evidence that the improvements come from the pretraining objective rather than simply the architecture.

  3. The experiments show that selective prediction improves robustness against different types of noise. For example, Tab. 2 shows that CORTEX-SPOT achieves the best performance among frozen EEG FMs on all 8 tasks at 12 dB muscle-band contamination.

  4. The authors also investigate why the proposed objective works. Fig. 2 shows that reconstruction allocates much more capacity to context-unpredictable noise, while Tab. 6 shows that reconstruction representations contain 11.9 times more probe-recoverable nuisance information than SPOT under re-injected noise. I think these experiments provide useful insights into what the model actually learns.

  5. I also find the selective prediction objective quite interesting. Instead of reconstructing waveform or spectral information, the authors force the model to predict discrete codes from the surrounding context. Although predictive learning is not new, I think investigating its effect on noise robustness for EEG representation learning is a meaningful direction, which could have an impact on future research in this field.

Critical weaknesses
  1. The authors should provide more ablation studies to show to what extent the proposed objective actually helps. For example, the authors did not investigate the contribution of the number of negative candidates, FSQ depth, or masking strategy. I think these experiments would help understand which design choices are important for the improvement.

  2. I suggest the authors investigate the variability across different pretraining runs. In Sec. 4.5, the authors use only one pretrained checkpoint for each objective, while the five seeds are only for downstream probes. Since the main contribution is about the pretraining objective, I think this is important to show whether the improvement is consistent.

  3. I also think the authors should investigate more carefully what kind of information is lost with selective prediction. The main hypothesis is that context-unpredictable information is less useful, but some clinically relevant EEG events might also be difficult to predict from surrounding context. I suggest the authors investigate whether the proposed objective can preserve these events, rather than only showing that less noise information is learned.

Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

R1.1 ablations and R1.2 variability.

Reply to Reviewer 1 (1/2): ablations and variability across pretraining runs

Thank you for the positive review and for the clear suggestions. We ran experiments for all three.

R1.1 Ablations of the number of negatives, FSQ depth and masking. We ran these, together with the temperature and the EMA decay. Each variant is a full 70-epoch pretraining of CORTEX-SPOT in which one setting changes: number of negatives (256, 1024, 4096), temperature (fixed 0.02 or 0.2, against the learned temperature of the main setting), FSQ depth (1, 3, 5), EMA decay (0.99, 0.999, 0.9999), and masking (uniform tube, random cells, mask ratio 0.5, against our variance-weighted tube). The table is in our shared-results comment ("Shared results: ablations and pretraining seeds") and in the appendix section "Ablations of the SPOT objective", with a pointer from Section 4.1.

On the three settings you named (number of negatives, FSQ depth and masking):

  • Number of negatives: performance on clean TUH signals is 63.3, 66.9 and 71.1 for 256, 1024 and 4096. More negatives improve performance on clean signals.
  • FSQ depth: depth 3 is 7.4 points lower than the main depth in performance on clean TUH signals.
  • Masking: with the reconstruction arm's random-cell masking, SPOT is still ahead of reconstruction at 12 dB on all three tasks (TUH 63.8 against 59.4, HMC 49.5 against 47.7, arousal 52.3 against 41.6). The variance-weighted tube raises performance on clean TUH signals by about 5 points over random-cell masking, but the 12 dB advantage does not depend on it.

Overall, the main setting is best or tied in performance on clean TUH signals, and the SPOT variants are ahead of reconstruction at 12 dB in 32 of 36 comparisons. So the improvement does not depend on one value of one setting. Each variant is one pretraining run, so differences below about 3 points should not be read as effects.

R1.2 Variability across pretraining runs. You are right that the five seeds in Section 4.5 vary only the probe. We pretrained two more CORTEX-recon and two more CORTEX-SPOT models with new random seeds, so each objective now has three pretraining runs, and we probed each with three downstream seeds. This gives nine comparisons per noise level. The table is in the shared-results comment.

  • On clean signals, the two objectives perform about equally (-1.0 +- 1.6 points).
  • Under 12 dB EMG, SPOT is ahead in all nine comparisons (+8.7 +- 5.0). At 6 dB and 0 dB it is ahead in eight of nine. The exception in both cases is one TUH seed.
  • The gap is most consistent for arousal at 12 dB (+11.0 to +12.7 in every seed) and for HMC at 0 dB (kappa 17.6, 9.7 and 23.3 for SPOT against 0.8, 0.1 and 0.9 for reconstruction).
  • For TUH, the claim is at 12 dB, where SPOT is ahead in all three seeds (+12.2, +2.7, +14.6).

The significance tests in the main text are over probe seeds on one checkpoint, as the paper states. For LaBraM we also pretrained two more seeds per arm, probed once each (appendix section "Variation across pretraining runs on LaBraM").

AuthorsOpenReview ↗

R1.3 what information is lost.

Reply to Reviewer 1 (2/2): what information is lost

R1.3 Are clinically relevant events preserved? Thank you. This is the right test of our hypothesis, and we ran it. The risk you describe is real in principle: an objective that keeps what the context can predict could drop events that the context does not announce. We therefore counted how many real events each frozen representation can still detect. We used the same probes as in the main experiments (three probe seeds, no retraining) on the held-out test sets.

How we count events:

  • The probes make one prediction for each short window of the recording. An event (an arousal or a seizure) is a run of consecutive windows that are labelled positive.
  • For each probe we set the detection threshold so that 5% of the windows without an event are flagged. Both objectives therefore pay the same false-alarm cost.
  • An event counts as detected if the probe flags at least one window that overlaps with it. We apply this same rule to both objectives.
Events detected (%)ReconstructionSPOT
All arousal events55.7 +- 4.074.8 +- 0.8
Abrupt-onset arousals76.495.8
Seizure events (TUH)66.3 +- 2.868.2 +- 2.4

Events that the preceding signal does not announce. Abrupt-onset arousals are the third of arousals with the largest rise in signal power compared with the 30 seconds before them. The signal before them gives little warning, so they are the closest match to the case you raised. SPOT detects 95.8% of them, against 76.4% for reconstruction. Because these events are strong, they are easier for both encoders than arousals overall; the point is that SPOT does not fall behind on them. Over all arousals SPOT detects 74.8% against 55.7%, and for seizures the two are equal (68.2% against 66.3%, within the spread across probe seeds).

Why these events are kept. "Predictable from context" does not mean "predictable from the past". The context of a masked cell also contains the cells after it and the other sensors at the same time, because each channel's masked span is drawn independently. An arousal or a seizure appears on several channels at once and lasts several seconds, so it is predictable from this context even when its onset cannot be foreseen. Noise that is local to one channel and does not recur in a consistent form does not have this property.

What we did not test. The argument is weakest for rare events that appear on a single channel, such as isolated epileptiform discharges. Our test sets do not label them, and the paper now states this as a limitation. The results are in the appendix section "Events detected from the frozen embeddings".

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of changes (to all reviewers)

Summary of changes (to all reviewers)

Thank you for your reviews. For each main request we ran new experiments and updated the PDF. Each reviewer has a reply below. Results that several reviewers asked about (ablations and pretraining seeds) are in the next comment, and the replies point to it.

Our claim. SPOT gives frozen representations that are more robust to contamination, while its performance on clean signals is comparable to reconstruction.

What changed in the PDF

  • The abstract, introduction and conclusion state the claim above.
  • Sections 4.1 to 4.5 are rewritten in plainer wording.
  • Section 4.5 now includes a test that separates the source of the target codes from the loss (2x2), with the full results in the appendix.
  • New appendix sections, with pointers from the main text:
    • ablations of the SPOT settings (negatives, temperature, FSQ depth, EMA decay, masking);
    • two more pretraining seeds for each CORTEX objective;
    • negative-source tests and identity probes;
    • a synthetic model with EMA-teacher targets;
    • which clinical events SPOT keeps;
    • FACED and BCIC-IV-2a;
    • a description of the two LaBraM arms.

Conventions used in our replies. "12 dB EMG" means muscle-band noise added to the test recording at a signal-to-noise ratio of 12 dB. All scores are on a 0 to 100 scale: AUPRC for seizure and arousal detection (TUH, Siena, HSP arousal), Cohen's kappa for sleep staging (HMC, Sleep-EDF, HSP staging), and accuracy for FACED and BCIC-IV-2a. Differences are in points on this scale.

AuthorsOpenReview ↗

Shared results: ablations and pretraining seeds (to all reviewers)

Shared results: ablations and pretraining seeds

1. Ablations of SPOT (asked by R1, R2 and R3). We trained one-factor-at-a-time variants of CORTEX-SPOT, each a full 70-epoch pretraining with the other settings at the main configuration. The table gives TUH (AUPRC), HMC (kappa) and HSP arousal (AUPRC). CORTEX-recon is the mean of three pretraining seeds.

VariantTUH, clean signalsTUH, 12 dBHMC 12 dBArousal 12 dB
Main SPOT setting71.170.451.654.4
256 negatives63.363.544.751.0
1024 negatives66.958.948.754.3
Temperature 0.02 (fixed)68.469.051.154.0
Temperature 0.2 (fixed)71.569.948.852.0
FSQ depth 165.664.148.152.9
FSQ depth 363.763.849.353.6
EMA decay 0.9961.458.239.648.7
EMA decay 0.99963.965.450.549.9
Mask ratio 0.564.263.748.454.2
Random-cell masking65.763.849.552.3
Uniform tube masking67.767.153.654.7
CORTEX-recon (3 seeds)70.759.447.741.6
  • The main setting is best or tied in performance on clean TUH signals. The SPOT variants are ahead of reconstruction at 12 dB in 32 of 36 comparisons (12 SPOT models on 3 tasks) and at 0 dB in 25 of 36.
  • The four comparisons where a variant is behind at 12 dB are: 256 negatives on HMC; 1024 negatives and EMA decay 0.99 on TUH; EMA decay 0.99 on HMC. Each of these variants also performs below the main setting on clean signals.
  • The largest drops in performance on clean TUH signals come from the EMA decay (-9.7 and -7.2), 256 negatives (-7.8), FSQ depth 3 (-7.4) and a mask ratio of 0.5 (-6.9).
  • With the reconstruction arm's random-cell masking, SPOT is still ahead of reconstruction at 12 dB on all three tasks.
  • Each variant is one pretraining run, so differences below about 3 points should not be read as effects. The full table (also 0 dB, and performance on clean HMC and arousal signals) is in the appendix section "Ablations of the SPOT objective".

2. Variability across pretraining runs (asked by R1). We pretrained two more CORTEX-recon and two more CORTEX-SPOT models with new random seeds, so each objective has three pretraining runs. We probed each with three downstream seeds. This gives nine comparisons per noise level (TUH, HMC and arousal, times three seeds).

ConditionSPOT minus reconstruction (mean +- SD, 9 comparisons)SPOT ahead in
Clean signals-1.0 +- 1.62 of 9
12 dB EMG+8.7 +- 5.09 of 9
6 dB EMG+13.8 +- 8.38 of 9
0 dB EMG+7.4 +- 10.68 of 9

On clean signals the two objectives perform about equally. The exception at 6 dB and 0 dB is one TUH seed (-3.7 and -11.0). Per-seed numbers are in the appendix section "Variation across pretraining runs". For LaBraM we also pretrained two more seeds per arm (one probe seed each); SPOT is ahead at 12 dB in all 15 task-by-seed pairs outside apnea, which is at the floor for both (appendix section "Variation across pretraining runs on LaBraM").

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.