- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SleepXA uses pretrained models for EEG and ECG along with a small network for remaining channels to create a sleep foundation model applicable to polysomnography data. The model was trained on thee datasets: SHHS, MESA and CFS totalling 9230 PSGs. The model is trained in stages unfreesing layers slowly aligning the representation of EEG, EKG and remainder. The modalities are aligned into a shared representation, and sleep-stage labels are used during pretraining. The authors evaluate the model on 27 tasks including sleep staging sleep apnea, disease outcomes and non sleep tasks. The paper reports top-2 on 22 of 27 tasks.
Critical strengths
-
Good set of tasks (27 diverse tasks including canonical sleep, disease, an other biosignal tasks) on a large pretraining dataset. The model is pretrained on 9,230 PSGs (a decent size although it could be made a lot bigger, specially for pretraining purposes) from three cohorts and evaluated on several external cohorts, with both linear probing and fine-tuning.
-
The paper aims to solve an important problem with modelling different modalities into a shared space based on occuring at the same time by applying losses that allow the model to disentangle cohort, sleep stage and modality from this main loss.
The manuscript is overall well written and the figures a
Critical weaknesses
- reliability of the evaluations (overstating performance):
- Some of the SleepFM results are so far off what is reported in the SleepFM paper itself (https://www.nature.com/articles/s41591-025-04133-4 fig3) compared to Table A.10 composit scores.
- The sleep staging label creeps in at pretraining time esdsnetially allowing the representations of the proposed SleepXA to be more aligned with the task and making these results less reliable.
- Unclear the impact of the different loss components
- The InfoNCE alignment is a variant of SleepFM's objective. CLIP-style alignment trails SleepXA by only 0.2–1.2 MF1 under LP (Table A.14).
- Therefore its unclear if the added losses actually help meaningfully on pretraining
Questions for authors
Could you add a clear analysis on how the different losses contribute?
Why is your reported SleepFM results underperforming the published values so much?
What is the effect on the sleep staging performance based on the sleep stage-based loss?
8 comments
Response to Reviewer KjKF (Part 2)
W2 / Q2: Clarification of the SleepFM evaluation protocol
W2: The reported SleepFM results are substantially lower than its published results, raising concerns about baseline evaluation and the reliability of the performance comparisons.
Q2: Why does SleepFM underperform its published results by such a large margin in your evaluation?
Response.
Thank you for highlighting this discrepancy. We recognize SleepFM as a pioneering contribution to sleep foundation modeling and disease prediction. Our reported results are a standardized re-evaluation of SleepFM, not a reproduction of its original Nature Medicine experiment[1]. We would like to clarify an important distinction between our evaluation protocol and that used in the original study.
In our study, SHHS contributes to PSG alignment, while disease evaluation uses a separate paired SHHS1–SHHS2 cohort comprising approximately 1,000 participants in total. We use pooled frozen features and a linear prediction head without demographic covariates. All compared models are evaluated using this same smaller downstream cohort and protocol.
By comparison, SleepFM was pretrained on large external cohorts, including the Stanford Sleep Clinic cohort, allowing SHHS to be reserved for downstream adaptation and evaluation. Its original SHHS evaluation used 3,291 training participants and LSTM-based prediction with age and sex [1]. Our study does not have access to a comparable external clinical pretraining cohort. The available downstream training data, prediction heads, covariates, outcome definitions, and evaluation metrics therefore differ. These differences provide important context for the relatively modest disease-prediction scores observed across models in our benchmark.
SleepMaMi provides direct evidence of the effect of the probing protocol: SleepFM’s staging MF1 increases from 56.3% under linear probing to 78.0% with an LSTM classifier [2]. We conducted a similar evaluation on Test SHHS1/2, where LSTM probing also improved SleepFM’s sleep-staging performance (Table R17).
Table R17. Effect of the probing protocol on Test SHHS1/2 sleep staging.
| Model / probing protocol | Acc. (%) | MF1 (%) | κ |
|---|---|---|---|
| SleepFM | 75.22 | 68.05 | 0.667 |
| SleepFM + LSTM | 83.67 | 76.21 | 0.772 |
| SleepFM + LSTM + age/sex | 83.00 | 76.41 | 0.766 |
| SleepXA | 84.07 | 76.24 | 0.780 |
| SleepXA + LSTM | 86.19 | 79.90 | 0.807 |
For disease prediction, using an LSTM head with age and sex improves performance, although scores remain modest, potentially reflecting the smaller cohort and limited case numbers.
Recent independent studies, including sleep2vec (ICLR 2026), OSF (ICML 2026), Omni-Sleep (MICCAI 2026), and SleepMaMi (ICML 2026), have also reported lower SleepFM scores in some downstream settings than those reported in the original Nature Medicine study [2–5]. These findings are broadly consistent with our observations, and the performance differences may partly reflect variations in datasets and evaluation protocols.
We will clarify these distinctions in the revised manuscript, and open-source our experimental settings for fair benchmarking.
References
[1] R. Thapa et al. A multimodal sleep foundation model for disease prediction. Nature Medicine, 32:752–762, 2026.
[2] K. Park, Y. Na, Y. R. Choi, H. Ryu, H.-W. Shin, and H.-S. Kim. SleepMaMi: A Universal Sleep Foundation Model for Integrating Macro- and Micro-structures. ICML, 2026.
[3] W. Yuan, Z. Jin, Y. Wang, D. Xie, Z. Ye, C. Zhang, and X. Chen. sleep2vec: Unified Cross-Modal Alignment for Heterogeneous Nocturnal Biosignals. ICLR, 2026.
[4] Z. Shuai, Z. Xu, D. Yang, W. Wang, and Y. Yang. OSF: On Pre-training and Scaling of Sleep Foundation Models. ICML, 2026.
[5] Z. Hou, S. Wang, K. Lou, M. Wang, C. Wei, and Q. Liu. Omni-Sleep: A Sleep Foundation Model via Hierarchical Contrastive Learning of CNS–ANS Dynamics. MICCAI, pp. 439–448, 2026.
Response to Reviewer KjKF (Part 3)
W3 / Q3: Contribution of sleep-stage guidance
W3: Using sleep-stage labels during pretraining may favor SleepXA on downstream sleep staging and complicate comparisons with models trained under different supervision settings.
Q3: How much does sleep-stage guidance contribute to sleep-staging performance, and how does performance change when this guidance is removed?
Response.
We thank the reviewer for raising this point. Sleep-stage guidance is an explicit feature of SleepXA, providing a physiological reference for aligning independently pretrained unimodal encoders. We appreciate the difficulty of acquiring such labels, especially training on large-scale label-free sleep recordings. Therefore, in line with our responses to the other two reviewers, we provide several alternatives reducing the reliance on the “full” sleep stage label, and therefore assess its contribution.
The experiments were conducted using four settings: (i) Full uses expert labels from SHHS, MESA, and CFS; (ii) None removes stage-dependent objectives while retaining instance contrastive learning, reconstruction, and other objectives that do not require sleep-stage labels; (iii) Hybrid uses expert SHHS labels and pseudo-labels for MESA/CFS generated by a model trained only on SHHS; and (iv) Partial uses expert SHHS labels while MESA/CFS contribute only to objectives that do not require stage labels.
As shown in Table R18, removing guidance reduces WSC staging MF1 by 1.17/1.26 percentage points and accuracy by 1.45/0.55 percentage points under LP/FT, respectively. Hybrid and partial guidance retain much of the full-label performance, with LP MF1 reaching 75.89% and 75.57%, versus 76.09% with full guidance.
Table R18. Effect of sleep-stage guidance on WSC sleep staging.
| Mode | Guidance | Acc. (%) | MF1 (%) | κ |
|---|---|---|---|---|
| LP | None | 83.71 | 74.92 | 0.751 |
| LP | Hybrid | 84.69 | 75.89 | 0.762 |
| LP | Partial | 84.23 | 75.57 | 0.760 |
| LP | Full (SleepXA) | 85.16 | 76.09 | 0.765 |
| FT | None | 85.40 | 76.35 | 0.770 |
| FT | Hybrid | 85.51 | 77.19 | 0.774 |
| FT | Partial | 84.90 | 76.20 | 0.765 |
| FT | Full (SleepXA) | 85.95 | 77.61 | 0.780 |
Table R19. Effect of sleep-stage guidance on WSC AHI estimation and OSA severity classification. Acc. and MF1 evaluate OSA severity classification.
| Mode | Guidance | MAE | RMSE | R² | Acc. (%) | MF1 (%) |
|---|---|---|---|---|---|---|
| LP | None | 3.655 | 6.584 | 0.853 | 73.41 | 73.68 |
| LP | Hybrid | 3.489 | 6.356 | 0.863 | 78.31 | 73.88 |
| LP | Partial | 3.508 | 6.556 | 0.854 | 77.51 | 73.73 |
| LP | Full (SleepXA) | 3.441 | 6.341 | 0.864 | 79.12 | 75.05 |
| FT | None | 3.621 | 6.611 | 0.852 | 78.72 | 75.19 |
| FT | Hybrid | 3.280 | 6.492 | 0.857 | 80.72 | 77.83 |
| FT | Partial | 3.410 | 6.508 | 0.856 | 80.31 | 77.31 |
| FT | Full (SleepXA) | 3.120 | 6.008 | 0.878 | 82.73 | 78.46 |
Table R20. Effect of sleep-stage guidance on CinC2017 AF recognition. All scores are percentages.
| Mode | Guidance | Challenge-F1 | AF-F1 | AF-AUROC |
|---|---|---|---|---|
| LP | None | 59.51 | 60.24 | 87.40 |
| LP | Hybrid | 72.91 | 73.35 | 89.60 |
| LP | Partial | 72.60 | 72.12 | 88.60 |
| LP | Full (SleepXA) | 73.18 | 74.30 | 91.15 |
| FT | None | 82.69 | 81.85 | 98.30 |
| FT | Hybrid | 82.71 | 81.56 | 98.20 |
| FT | Partial | 82.39 | 81.41 | 98.10 |
| FT | Full (SleepXA) | 83.35 | 82.30 | 98.40 |
Removing guidance also reduces OSA classification accuracy by 5.71/4.01 percentage points under LP/FT and LP AF-F1 by 14.06 percentage points (Tables R19 and R20). The AF-F1 gap narrows to 0.45 percentage points after FT. These results suggest that stage guidance supports transferable representations beyond improving sleep staging alone. This task-dependent pattern is consistent with the smaller gap between SleepXA and CLIP-style alignment on staging and the larger advantage on AF recognition.
These results demonstrate that sleep stage labels are important, but do not necessarily need to cover the whole training set for pretraining. Although full sleep-stage guidance achieves the best performance, alternative solutions such as pseudo-label and partial-label guidance can achieve comparable performance to the full version, when expert annotations are prohibitive to acquire.
We hope these results and clarifications address your concerns and support a more positive assessment of our work. We would be happy to answer any further questions.
Response to Reviewer KjKF (Part 1)
We are grateful for the reviewer's attentive analysis and helpful feedback.
W1 / Q1: Contributions of the learning objectives
Response.
We thank the reviewer for this question. The additional objectives implement sleep-guided cross-modality common–specific learning for composing independently pretrained physiological encoders. They serve complementary roles: common-space alignment encourages cross-modal correspondence within matched cohort–stage conditions; stage-conditioned specific learning preserves complementary modality-dependent information; and conditional residual disentanglement penalizes redundancy after removing condition-specific means. Reconstruction additionally encourages information preservation.
The complete objective achieves the highest LP F1 on all three evaluated tasks: WSC sleep staging, WSC OSA severity classification, and CinC2017 AF recognition (Tables A.18–A.20). The LP gain over CLIP-style alignment is 0.83 percentage points for WSC sleep staging (Table A.16, formerly Table A.14) and 19.75 points for AF recognition (Table A.17; Table R16 below). Thus, sleep-staging performance alone does not fully reflect the benefit of the combined objectives for cross-context transfer. This pattern is also consistent with the stage-guidance experiments, whose contribution we discuss separately in our response to W3.
Table R16. Cross-context transfer on the CinC2017 AF task. AF-F1 scores (%) are reported under linear probing (LP).