acceptodds
Withdrawn from ICLR 2027

Cross-Modal Pretraining with Modality-Specific Risk Prediction from Polysomnography

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Polysomnography (PSG) records multiple physiological systems overnight and may contain prognostic information beyond sleep diagnosis. Using these signals for future disease prediction is challenging because PSG modalities differ in signal structure, availability, and relevance to individual outcomes. Although cross-modal learning can exploit shared physiology, merging all modalities into a single downstream representation may obscure information specific to individual physiological systems. We introduce a multimodal self-supervised framework that separates cross-modal pretraining from downstream risk integration. Independently parameterized encoders are trained using within-modality masked latent prediction and cross-modal prediction from other simultaneously observed signals. For time-to-event prediction, the encoders are frozen, and endpoint-specific whole-night aggregators and risk heads estimate risk separately from each available modality. These estimates are combined only at the prediction stage, allowing inference from incomplete modality sets without imputing missing signals or representations. We evaluate ten incident disease outcomes in 26,782 participants from three development cohorts and six harmonized outcomes in an external cohort of 1,000 participants. In pooled evaluation, combining PSG with basic clinical covariates achieves a mean Harrell’s C-index of 0.717. Although selected baseline performs better under the matched frozen probe, endpoint-specific downstream modeling increases the mean C-index of our PSG representation from 0.681 to 0.696 and yields higher point estimates than the SleepFM probe for eight of ten outcomes. Representation similarity exceeds the permutation null for 14 of 15 modality pairs, yet the predictive contribution of each modality varies across endpoints. Leave-one-cohort-out, external-cohort, and modality-removal evaluations further assess transfer under heterogeneous input configurations. These findings show that multi-modal PSG can benefit from cross-modal learning without requiring physiological modalities to share a single downstream representation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Polysomnography, Self-supervised learning, Multimodal representation learning, Disease risk prediction, Cross-cohort generalization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.20 over 5 reviews, above 2% of the others.
Reviewer_FB6wOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a multimodal self-supervised framework that decouples cross-modal pretraining from downstream risk integration for polysomnography (PSG) and verifies its effectiveness on a large-scale dataset. Results suggest that the framework benefits from cox and have complementary effect to demographic signals. However, the experimental design provides insufficient evidence to validate the effectiveness of individual components, while the limited baselines and presentation issues further weaken the paper.

Critical strengths
  1. The paper presents a novel self-supervised learning design that explicitly models both within-modality and cross-modal information, while naturally handling missing modalities without requiring imputation.
  2. The experiments are conducted on a large-scale dataset spanning multiple cohorts and disease endpoints, providing relatively comprehensive empirical validation.
  3. Some analyses are interesting. The analysis of cross-modal representation similarity is interesting and helps characterize the relationships among heterogeneous physiological signals. In addition, Fig. 4 provides a useful study of the incremental contribution of PSG signals beyond basic clinical covariates.
Critical weaknesses
  1. The reported results suggest that PSG provides very small benefit for several endpoints. While the clinical fusion weight ww is selected on the validation set, it would be useful to report the selected ww for each endpoint and quantify how much the final performance actually depends on PSG versus the clinical covariates.
  2. The comparison with SleepFM is not fully controlled. The paper compares SleepFM + frozen Cox against Ours Full PSG, where the latter additionally uses endpoint-specific whole-night aggregators and modality-specific risk heads. Since SleepFM already performs better under the matched frozen Cox probe (0.692 vs. 0.681), it is unclear whether the gain of the full model comes from a better representation or simply from using a stronger, separately trained downstream head for each task.
  3. The proposed SSL formulation combines within-modality masked prediction and cross-modal prediction, but there is no clear ablation isolating the contribution of each SSL objective.
  4. The final PSG risk is obtained by simply averaging the available modality-specific Cox risk scores. A learned or validation-selected modality weighting scheme would seem more appropriate.
  5. The baseline coverage is limited. Beyond SleepFM, the paper would benefit from comparisons with more recent multimodal/self-supervised sleep representation models, such as sleep2vec (sleep2vec: Unified Cross-Modal Alignment for Heterogeneous Nocturnal Biosignals, 2026).
  6. The presentation needs substantial polishing. There are unresolved or malformed citations (e.g., “(?)” and “(??)” in Sec. 2.2), duplicated/redundant paragraphs, and some figures use very small fonts that are difficult to read. These issues noticeably reduce the clarity and professionalism of the submission.
Questions for authors
  1. Could you report the selected ww for each endpoint and its impact on performance?
  2. Could SleepFM be evaluated with the same endpoint-specific aggregators and modality-specific risk heads?
  3. Sleepfm2 is released, although SleepFM2 is concurrent work, could the authors briefly discuss its differences from this method?
Reviewer_E2m4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a self-supervised method that uses Polysomnography (PSG) to predict future disease. Six signal types (brain, heart, breathing, oxygen, pulse, muscle) has its own encoder. In pretraining, each encoder learns by filling in masked parts of its own signal and by predicting its representation from the other signals. To predict each disease, the encoders are frozen, and each signal type gives its own risk score. The scores from the available signals are averaged, so it is free from missing signal problem. The model is trained and tested on three Human Sleep Project cohorts and also tested on part of the SHHS. Across ten diseases, the full PSG model reaches a mean C-index of 0.696, compared with 0.681 for its own frozen features with a simple Cox model and 0.692 for SleepFM with the same simple model. Adding age and sex raises the mean to 0.717. The paper also studies how similar the signal representations are, what happens when signals are removed, and leave-one-cohort-out results.

Critical strengths
  1. The problem matters. PSG captures several body systems, and using routine sleep studies to predict future disease is a useful goal.
  2. The design is sensible. Shared pretraining can learn what the signals have in common, while separate risk scores can keep information specific to each signal. Averaging scores is a simple way to handle missing signals.
  3. Broad evaluation. It covers many diseases, an external cohort, cohort transfer, and signal removal, with confidence intervals and paired bootstrap tests.
Critical weaknesses
  1. The value of cross-modal pretraining is not shown directly. There is no comparison with the same model trained without the cross-modal loss, or with other cross-modal objectives. Without this, the main claim is not fully supported.

  2. The main comparison is not fair. SleepFM gets only a simple frozen Cox model, while the proposed model gets disease-specific aggregators and risk heads. When both use the same simple model, SleepFM does better (0.692 vs. 0.681). So the gains may come from the larger downstream model, not from better representations.

  3. Test data may have been used in pretraining. The paper says held-out cohorts may have contributed unlabeled data to pretraining. The authors should state clearly whether the test participants and the LOCO held-out cohort were used to train the encoders. If so, the results do not show true generalization to unseen data.

  4. The clinical baseline is weak. Clinical baseline using age and sex alone are not enough to claim that PSG adds value. A stronger clinical model would help to make the main results concrete.

  5. The paper looks unfinished. It has missing citations (“?”), a broken appendix reference (“Appendix ??”), a repeated paragraph in the introduction, and references to appendix sections that don’t appear. This makes some details hard to check. Finally, it exceeds the submission page limit, which should be desk-rejected.

Questions for authors

See weakness

Reviewer_25GkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a multimodal polysomnography (PSG) framework for disease-risk prediction, using six independently parameterized modality encoders pretrained through within-modality masked latent prediction and cross-modal prediction with exponential-moving-average teacher targets. During downstream training, the encoders are frozen, and endpoint-specific aggregators and risk heads generate modality-level risk scores, which are averaged and optionally integrated with an age-and-sex Cox model. The framework is evaluated on ten incident outcomes using 26,782 participants from three development cohorts, with additional leave-one-cohort-out evaluations, missing-modality experiments, representation analyses, and external evaluation on 1,000 SHHS participants. The reported mean C-index reaches 0.696 for the full PSG model and 0.717 with demographic integration. However, under the matched frozen Cox probe, SleepFM outperforms the proposed representation, achieving a C-index of 0.692 compared with 0.681.

Critical strengths
  1. The problem is relevant and practically motivated. Learning from heterogeneous PSG recordings while accommodating incomplete modality sets is useful. The development dataset is substantial, and evaluating several incident outcomes provides a broader perspective than a single-endpoint study.
  2. The framework is coherent at a high level. Separating representation learning from endpoint-specific survival modeling provides a reasonable modular design. The cross-modal objective explicitly excludes the target modality from its own context, and downstream prediction can operate on observed modalities without imputing missing signals.
  3. Several evaluation choices are appropriate. The manuscript describes participant-level partitions, exclusion of pre-existing diagnoses, a 30-day landmark, training-derived feature normalization, validation-selected clinical fusion, and participant-level bootstrap uncertainty estimates.
  4. The authors acknowledge important limitations. In particular, they report that SleepFM performs better under the matched frozen probe and clarify that the LOCO experiments assess downstream transfer because held-out cohorts may have contributed unlabeled pretraining data.
Critical weaknesses

W1. Unconvincing performance improvements and unfair baseline comparison.

Under the matched frozen Cox probe, SleepFM outperforms the proposed method (0.692 vs. 0.681). The proposed model only surpasses SleepFM after introducing more sophisticated downstream components. Without applying comparable downstream architectures and clinical fusion to SleepFM, the claimed improvement is not convincing.

W2. Missing critical ablation studies.

The paper does not isolate the contribution of cross-modal pretraining. Essential comparisons, such as removing the cross-modal loss or auxiliary unimodal objectives, are missing. Furthermore, the proposed fusion strategy shows only marginal gains in Table 2 and performs worse on LOCO-I0004.

W3. Inconsistent experimental configurations.

Figure 3 reports 30% modality dropout, whereas Table S4 states that no downstream modality dropout was used. The paper does not clarify which configuration produced the reported results, raising concerns about experimental reproducibility and comparison fairness.

W4. Numerical inconsistencies in reported results.

Section 5.4 reports a C-index improvement of +0.113 for cardiomyopathy, while Table 1 implies +0.097. Similarly, the reported +0.091 improvement for type 2 diabetes differs from the +0.097 implied by Table S3. These discrepancies require clarification.

W5. Insufficient reproducibility and poor manuscript preparation.

Key implementation details and hyperparameter settings are missing. The manuscript also contains unresolved citations (“?”, “??”), duplicated paragraphs, and unremoved template instructions. The abstract is divided into three paragraphs, and Figure 1 contains excessively small labels. These issues suggest that the manuscript has not been thoroughly checked before submission.

Questions for authors
  1. How does the full model compare with an otherwise identical model pretrained using only the within-modality objective, with λc=0\lambda_c = 0? What is the effect of removing the auxiliary unimodal Cox losses?
  2. What results are obtained when SleepFM receives a comparably expressive endpoint-specific downstream head and the same age-and-sex integration? For Table 2, are the aggregators, parameter counts, training losses, and tuning budgets matched across fusion strategies?
  3. Which results use 30% downstream modality dropout, and which use no downstream modality dropout? Is Figure 3 based on the same checkpoint configuration as Tables 1 and 2, and does the robustness advantage persist under matched dropout training for the baseline?
  4. What explains the cardiomyopathy and type 2 diabetes improvement discrepancies between Section 5.4 and Tables 1/S3? Were different populations or models used? Why does Figure 4 evaluate a different endpoint set from the primary analysis?
Reviewer_k6VROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper develops a model on PSG data to predict clinical outcomes for 10 diseases. The model achieves C index of around 0.717. The work is quite similar to previously published works such as sleepFM and is smaller in scale and in validations.

Critical strengths

Modeling multi-modal PSG signals is interesting and being able to predict downstream clinical applications can have useful applications.

Critical weaknesses

This is an application oriented paper and the methodological innovations are relatively limited (which is fine for applications). So I place more emphasis on the rigor of the experimental design and comparisons, where this paper has several limitations.

First, comparisons to SleepFM are not apples-to-apples. With frozen encoders, both representations scored similarly (SleepFM slightly higher on some indications). The other comparisons with SleepFM + linear probe vs the author's trained model don't isolate the benefit of the representation learning. Moreover, the improvements are quite small for most of the indications with overlapping confidence intervals.

Detailed ablations to demonstrate the benefit of the author's pretraining is also missing.

Questions for authors

Provide ablations and stronger evidence to isolate the benefit of the pretraining.

Reviewer_H1hYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a multimodal self-supervised learning framework that uses polysomnography (PSG) signals to predict future disease risks. The main idea is to learn shared information across different physiological signals during pretraining, while preserving modality-specific information for downstream prediction. The authors use separate encoders for six physiological modalities and train them with within-modality masked prediction and cross-modal prediction. For each disease, the pretrained encoders are frozen, and modality-specific aggregators and risk heads are trained. The risk scores from available modalities are then averaged, allowing the model to handle missing modalities without imputation.

The authors evaluate their approach on ten disease outcomes across three development cohorts, with additional evaluation on an external cohort. They compare against SleepFM and simpler feature-fusion approaches, and examine representation similarity, missing-modality robustness, and cross-cohort generalization. Their full PSG model achieves a mean C-index of 0.696, compared with 0.692 for the SleepFM frozen Cox baseline. Combining PSG predictions with age and sex further improves the mean C-index to 0.717. The results also suggest that different modalities share some information but contribute differently to predicting specific diseases.

Critical strengths
  1. Practical motivation and reasonable framework. I think the paper addresses an interesting and practically relevant problem. Different PSG modalities capture different physiological information, and not all modalities are available across patients or cohorts. The idea of learning across modalities during pretraining while keeping modality-specific pathways for downstream prediction is reasonable. I particularly appreciate that the framework can handle missing modalities without requiring imputation.

  2. Relatively comprehensive experimental evaluation. I appreciate that the authors evaluate the framework across multiple disease outcomes and cohorts, including external validation and experiments on missing modalities. The results also suggest that PSG signals provide additional predictive information beyond age and sex, which is encouraging for potential clinical applications.

Critical weaknesses
  1. Insufficient evidence for the benefit of cross-modal pretraining. One of the main ideas of the paper is to learn shared information across physiological modalities while preserving modality-specific information. However, I did not find a direct ablation comparing the proposed pretraining objective with within-modality learning alone. Although the representation similarity analysis is interesting, it does not establish whether cross-modal pretraining actually improves downstream disease prediction. This makes it difficult to assess the contribution of one of the paper's central methodological components.

  2. Unclear sources of performance improvement and potentially unfair baseline comparisons. Under the same frozen Cox probe, the proposed representations perform worse than SleepFM (0.681 vs. 0.692). The authors' full model achieves a slightly higher C-index (0.696), but it also uses more sophisticated downstream aggregation and modality-specific risk heads. Without evaluating SleepFM with a comparable downstream architecture, it is unclear whether the improvement comes from the proposed representation learning or the downstream modeling choices. This weakens the evidence that the proposed framework offers a meaningful advantage over existing approaches.

Questions for authors
  1. Cross-modal pretraining: Could the authors clarify how much the cross-modal prediction objective contributes to the downstream performance? In particular, how does the model perform with only within-modality pretraining, keeping the downstream architecture unchanged? This would help me understand whether cross-modal learning is necessary for the reported improvements.

  2. Comparison with SleepFM: Although the paper includes a matched frozen Cox comparison, the full proposed model uses more sophisticated downstream aggregators and risk heads. Could the authors provide a comparison where SleepFM uses a comparable downstream architecture and the same clinical fusion procedure? This would help clarify whether the improvements come from the proposed pretraining method or the downstream modeling choices.

  3. Missing references and experimental details: I noticed several unresolved references, including (?) and (??) in Section 2.2 and Appendix ?? in Section 4.2. Could the authors clarify the intended references and where the complete hyperparameter search details can be found? In particular, I could not locate the full hyperparameter grids mentioned in Section 4.2, which makes it difficult to assess the model selection procedure and reproducibility.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.