- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes Temporal NT-Xent, a contrastive loss for windowed EEG. Negatives from the same patient are down-weighted in the softmax denominator by w = 1 − exp(−λd), where d is the distance between window indices. Cross-patient negatives keep weight 1 and the augmentation positive is unchanged. A ~131K-parameter STFT-CNN encoder is pretrained for 5 epochs on the training subjects of the OpenNeuro ds008108 sleep cohort (20-channel EEG, 30 s windows). It is then frozen and probed with an MLP for subject-level OSA vs. control classification, with window probabilities averaged per patient. On one patient-level split with 23 test patients, AUROC rises from 0.5417 (vanilla NT-Xent) to 0.7667 (p = 0.0455). Across five seeds, the mean gain is +0.035 ± 0.160 and A1 is better in 2 of 5 seeds. The paper also compares against a continuity regularizer, a PhysioCLR-style loss, an instance-wise SoftCLT adaptation and three weighting kernels. It further reports an effective-rank diagnostic, zero-shot transfer of the frozen encoder to SHHS-1 (AUROC 0.724 vs. 0.718 for a single-channel control), and INT8 quantization (71% smaller, encoder cosine similarity 0.9991).
Critical strengths
The question is well posed and the experimental design is the right kind for it. The paper asks where temporal information should enter a contrastive objective, and compares several answers under one shared encoder, augmentation, budget and probing protocol. Splitting is at the patient level, and the audit documents state that SSL pretraining uses only training subjects, which avoids the transductive leakage common in small EEG studies. This matters because a controlled comparison of this type would be useful to the field. The strength is conditional, though: I could not verify the split code, and some confounds between conditions (below) weaken the “controlled” label.
The authors are unusually open about negative results, and the archive includes per-patient prediction files with patient IDs. The paper reports that the in-domain gain is not seed-robust and that effective rank does not improve, rather than hiding these outcomes. The saved predictions and checkpoints let a reader recompute the reported metrics, which is how I found several of the problems below. This transparency is worth acknowledging even though it ends up exposing that the main claims are not supported.
Critical weaknesses
The paper's contribution is a pretraining objective, but the method is never used the way pretraining is used. The introduction motivates SSL by the "large volumes of unlabeled recordings but comparatively scarce clinical labels" in EEG. Yet the encoder is pretrained only on the 66 training subjects of a single cohort. These are the same labeled subjects later used to train the probe. That is 66 nights of EEG, roughly 450 hours if the training nights are as long as the test nights, and the encoder sees them for five epochs, with the loss still decreasing at the end (fig3b_ssl_training_curves.png). No unlabeled data beyond the labeled training set is used at any point. For comparison, SleepFM was pretrained on more than 100,000 hours of PSG (Thapa et al., ICML 2024). Even the remaining 31 subjects of ds008108, or SHHS itself, which the authors say they had access to, were not used for pretraining.
In this setup, the SSL step works more like a regularized feature extractor fitted to the training set than like pretraining. Results obtained this way say little about how the objective behaves at the scale where pretraining objectives matter. If Temporal NT-Xent is meant as a pretraining scheme, I would expect it to be pretrained on a large, multi-subject corpus and then transferred. Its comparison with vanilla NT-Xent should be made in that setting.
The small data regime also weakens the motivation itself. The premise is that windows recorded close together in the same night are partial false negatives and should repel each other less. The paper never measures this. The value λ = 0.1 (weight 0.63 at five minutes, 0.95 at fifteen) is not linked to sleep-stage bouts or to the 10 to 60 s duration of respiratory events.
The evaluation side is just as small. All in-domain conclusions come from one split with 23 test patients, 8 of them with OSA. With 120 positive-negative pairs, moving a single OSA patient from the top to the bottom of the ranking changes AUROC by 0.125. The headline gain of +0.225 has a paired interval of [0.05, 0.43] and shrinks to +0.035 ± 0.160 across seeds. None of the comparisons between temporal mechanisms in Sections 5.3 and 5.4 are significant across seeds, yet the paper draws general conclusions from them. The task is also narrow: subject-level OSA vs. control classification on one dataset, with no event detection or AHI estimation. That is far from the general claims in the title or the cross-cohort and deployment sections.
A pretraining objective should be compared with those used today. NeuroAtlas (Kontras et al., arXiv 2605.14698, 2026) benchmarks EEG and time-series foundation models with the same frozen-probe protocol used here. It covers 42 datasets, including sleep medicine and brain-age estimation, and finds that model rankings change considerably across datasets within a domain. That finding is the reason one 23-patient split is not enough. Evaluating A0 and A1 on its sleep tasks would show whether the weighting generalizes. Given how recent NeuroAtlas is, I see this as a comparison that should be considered rather than one the authors were required to run. Independently of it, the paper has no supervised baseline, no spectral-feature baseline, and no pretrained encoder such as BIOT, LaBraM (ICLR 2024), CBraMod (ICLR 2025), or SleepFM.
The novelty claim also needs to be discussed against prior work. StatioCL (Wu et al., CIKM 2024) already reweights negatives by temporal distance from the anchor, and Dependent TS2Vec (Marusov et al., arXiv 2025) builds temporal closeness into the contrastive loss. For biosignals, Cheng et al. (2020) use subject-specific negatives, CLOCS (Kiyasseh et al., ICML 2021) uses same-patient positives, and Banville et al. (J. Neural Eng. 2021) use temporal distance on sleep EEG as the training signal.
Beyond the scale problem, the released artifacts do not support the headline results. The archive contains no implementation of the loss, training, evaluation, SHHS, or quantization, only a dataset stub (sleep_apnea_dataset.py). The points below come from the checkpoints, prediction files, and the authors' own audits.
The five-seed analysis does not vary pretraining. The five A0 encoders in results/multiseed/seed_*/ are byte-identical (md5 eb67cec4…). So are the five A1 encoders (d09cc927…) and the five corrected A3 encoders in results/A3_multiseed/ (d1e499c4…). The audit traces this to a hard-coded set_seed(42) in pretrain_ssl() (REPRESENTATION_EFFECTIVE_RANK_AUDIT.md, lines 22 to 26). Tables 2, 7 and 8 therefore measure only downstream MLP variability, contrary to Appendix D and Section 7. A0 was never re-seeded, so there is no valid multi-seed A0 vs. A1 comparison.
The canonical result is also the most favorable of three seed-42 runs in the archive. Those runs give A0 vs. A1 of 0.600 vs. 0.550 (MEGA_EXPERIMENTAL_REPORT.md, §11), 0.600 vs. 0.725 (results/loss_ablation/), and 0.5417 vs. 0.7667 (results/final_primary/, the run used in the paper). Table 1's A3 value of 0.7250 comes from an implementation the authors' fix report says to discard (A3_IMPLEMENTATION_FIX.md, lines 84 to 87); the corrected value is 0.5917. In addition, according to the audit, A1 removes the positive from its denominator while A0 keeps it (A1_IMPLEMENTATION_AUDIT.md, line 251). The A0 vs. A1 comparison therefore changes two things at once.
I could not find evidence for the SHHS-1 result. The only cross-cohort file (cross_cohort_results.npz) is a Sleep-EDF wake/sleep run with AUROC 0.50. P0_FINAL_EXPERIMENTAL_STATUS.md (line 60) states that SHHS transfer was not evaluated. The paper also does not say how the 20-channel encoder was applied to SHHS, how OSA was defined there, or what the control is. Even taken at face value, 0.724 vs. 0.718 says nothing specific about Temporal NT-Xent.
The labels are strongly confounded with demographics. In participants.tsv, controls are younger (median age 24 vs. 34.5) and less often male (49% vs. 84%). On the 23 test subjects, which I recovered from the saved patient IDs, age alone gives AUROC ≈ 0.83 and age plus sex ≈ 0.84. Both are above every SSL result in the paper.
The quantization claims do not match the archive either. Appendix E describes static calibrated W8A8. CODEBASE_AUDIT.md (lines 70 to 76) describes dynamic weight-only quantization that was never run on patient data. The reported 0.7611 is not a possible AUROC with 8 positive and 15 negative patients, since attainable values are multiples of 1/240.
A few smaller points. The 66/22/23 split adds up to 111 subjects, not 142, and the exclusions are not explained. The effective rank of the same A0 encoder is 7.30, 8.08, or 10.51, depending on the file. Preprocessing lacks anti-aliasing. λ was never tuned. Finally, the A1 seed-42 probe assigns every test patient a probability above 0.59.
Questions for authors
-
Why was pretraining restricted to the 66 labeled training subjects, with no additional unlabeled data? A consistent advantage for A1 over A0 when both are pretrained on a substantially larger corpus would change my assessment the most. The corpus could be the remaining ds008108 subjects plus a public PSG cohort, with the frozen encoders evaluated on several downstream datasets, for example the sleep, epilepsy and brain-age tasks of NeuroAtlas.
-
The A0 and A1 encoders in results/multiseed/ are byte-identical across seeds. Were Tables 2, 7 and 8 produced with the hard-coded set_seed(42) in pretrain_ssl()? The archive also contains three seed-42 runs, and they disagree on the sign of the gain; why is results/final_primary/ the canonical one? Re-running A0 and A1 with the SSL seed actually varied, ideally over repeated patient-level splits with pretraining inside each fold, would address both points.
-
Does the A1 used for all reported results exclude the augmentation positive from its denominator, as the audit indicates, while A0 keeps it? If so, could you report decoupled NT-Xent without temporal weights? That would show how much of the A0 to A1 difference comes from the temporal weighting itself.
-
Please provide the code and prediction files for the SHHS-1 result, and describe the protocol in full. That means how the 20-channel encoder was applied to the SHHS EEG derivations, how OSA was defined, how many participants were included, which head produced the scores, and what the single-channel control is. How does this result relate to cross_cohort_results.npz, which is a Sleep-EDF wake/sleep run at chance, and to the statement in P0_FINAL_EXPERIMENTAL_STATUS.md that SHHS was not evaluated?
-
On the 23 test subjects, age alone reaches AUROC ≈ 0.83, above every SSL result in the paper. Could you report age and age-plus-sex baselines on the same split, along with an age- and sex-matched or covariate-adjusted evaluation of all objectives? Without this, it is unclear whether the encoders capture OSA-related EEG features at all.