- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SemPSG, a sleep foundation model that uses channel semantics to guide the integration of signals within each modality. It combines raw physiological signals with three image views: STFT spectrograms, CWT scalograms, and morphology maps. The model performs strongly across several downstream tasks, including a reported 9.6 percentage-point improvement in macro-F1 over the second-best baseline for sleep-disordered breathing classification. It also presents a novel task - cognition and affective assessment.
Critical strengths
- The evaluation covers a broad range of downstream tasks and shows strong performance relative to general-purpose time-series and sleep foundation models.
- Combining raw signals with image views offers a plausible way to capture complementary waveform, spectral, and temporal information.
Critical weaknesses
- The contribution of the main components remains difficult to assess. The channel-name shuffling visualization suggests that semantics affect the learned gating weights, but quantitative ablations are needed to establish how much channel semantics and each image view improve downstream performance. Comparisons with versions that omit semantic conditioning, omit image views, or use individual image views would help justify the added complexity.
- Several masked reconstruction objectives use intermediate embeddings as targets. Stop-gradient prevents gradients from flowing through a target within a training step, but the shared components producing that target may change in subsequent steps. The paper does not provide enough information to assess whether these moving targets affect training stability or risk representation collapse. Training diagnostics or an explanation of the stabilization strategy would strengthen this part of the method. For example, JEPA [1] or DINO [2] training uses several tricks to stabilize the training.
- The paper reports results for a single model size, so it is unclear whether performance improves as the model grows. Experiments across multiple model sizes, with training compute reported, would help establish whether SemPSG shows the scaling behavior observed in recent sleep foundation models [3, 4].
- The channel semantics tokenizer is a central contribution, but the paper does not show how it handles names absent from its pretraining data. An evaluation that holds out channel names or naming conventions would help determine whether it learns transferable physiological semantics rather than relying on familiar labels.
Questions for authors
- Image-view aggregation: The image views aggregate channels using fixed weights derived from window-level signal statistics. How exactly are these weights calculated? Why was this approach chosen instead of using the learned channel semantics for aggregation?
- Channel-semantics pretraining: How is () computed from the different semantic attributes, including missing or inapplicable labels? How is the direction loss handled for channels without an ordered derivation? Please also clarify how negative examples are selected for the alias and direction objectives - greater number of negative examples is important for contrastive learning to work.
- Semantic modulation: Equation (3) defines the modulated representation (), but the subsequent reconstruction and channel-aggregation equations use . Is () used only as the input to self-attention, with denoting the resulting contextualized representation? Please make this distinction explicit.
- Masking and information leakage: For the time-series reconstruction objective, are tokens masked before or after the convolutional tokenizer? If convolutional receptive fields cross patch boundaries, information from a masked patch might enter neighboring visible tokens. Please clarify the tokenizer’s receptive field and how the implementation prevents this leakage.
- Final representation: Why does the final representation retain separate time-series embeddings for each modality but aggregate the image views into a single embedding? What motivates this asymmetry, and how does it affect downstream performance?
Minor issues
- Please check the index and subscript formatting around lines 200 and 280. In Equation (12), the summation index also appears inconsistent with the indices in the loss term.
- The inline notation around line 237 is difficult to parse and would benefit from clearer formatting.
[1] Assran, Mahmoud, et al. "Self-supervised learning from images with a joint-embedding predictive architecture." 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023.
[2] Caron, Mathilde, et al. "Emerging properties in self-supervised vision transformers." 2021 IEEE/CVF international conference on computer vision (ICCV). IEEE, 2021.
[3] Shuai, Zitao, et al. "OSF: On pre-training and scaling of sleep foundation models." arXiv preprint arXiv:2603.00190 (2026).
[4] Carter, Jonathan F., and Lionel Tarassenko. "Next-Token Prediction Learns Generalisable Representations of Sleep Physiology." arXiv preprint arXiv:2606.09605 (2026).
6 comments
Author Response (Q1-Q5)
Q1: Channel aggregation in the image branch.
For each 30-second window, we normalize the standard deviations of valid channels within each physiological modality to obtain the channel weights:
, \epsilon = 10^{-6}.$$ Here, $\sigma_c$ is the channel standard deviation, $\mathcal{C}_m$ is the set of valid channel in modality $m$ and $\epsilon$ prevents division by zero. We chose this approach to generate image and reuse them during pretraining, avoiding repeated signal transformations. Semantic aggregation weights that change during training would require reaggregating channel-level feature maps after weights updates, preventing direct reuse of cached images. Thus, we use learned aggregation guided by channel semantics in the waveform branch. The image branch uses deterministic statistical aggregation to provide complementary time-frequency and morphological information. > **Q2: Channel semantics training objectives and negative sampling.** The pretraining objective is $L_{\mathrm{sem}} = L_{\mathrm{attr}} + L_{\mathrm{alias}} + \lambda_{\mathrm{dir}} L_{\mathrm{dir}}$, where $\lambda_{\mathrm{dir}}$ weights the direction loss. $L_{\mathrm{attr}}$ sums weighted prediction losses over four attributes (signal type, derivation, location, and sensor), computed strictly on known applicable labels while masking out missing or inapplicable ones. Channels without an explicit ordered derivation are similarly masked out from $L_{\mathrm{dir}}$. For alias contrastive learning, $L_{\mathrm{alias}}$ uses all eligible candidate names within the training split, providing about 261 negative examples per valid anchor on average. Verified aliases form positive pairs, while names with known semantic differences form negative pairs weighted by their semantic relationships, excluding uncertain pairs to reduce false negatives. $L_{\mathrm{dir}}$ uses triplets comprising an anchor name, an equivalent alias, and a name with reversed electrode order. Reversed negatives are taken from existing training names when available or generated by swapping electrode order, excluding any candidates that overlap with validation or test names. > **Q3: Use of semantically modulated representations.** In the following Equation, $\mathbf{\tilde{z}}$ is used to compute self-attention on the modulated representations. The $\mathbf{z}$ in the reconstruction and channel aggregation equations denotes the contextualized output after self-attention, residual connections, and feed-forward layers. Both operations use representations conditioned on channel semantics. The aggregation module also uses the semantic embeddings directly to compute its weights. $\widetilde{\mathbf{z}}^{(t)} _ {m,c,p} = \left(1 + \gamma^{(t)} _ {m,c}\right) \odot \operatorname{LN}\left(\mathbf{z}^{(t)} _ {m,c,p}\right) + \beta^{(t)} _ {m,c}.$ We will update the reconstruction and aggregation equations to distinguish it from the modulated attention input. > **Q4: Masking strategy and potential information leakage.** The mask strategy is applied after CNN tokenization and before context encoding. We first divide the waveform into 60 non-overlapping 0.5-second patches, then perform convolution and normalization on each patch independently, ensuring that information does not propagate to adjacent patches. The masked tokens are subsequently replaced with learnable mask tokens, while the unmasked tokens serve as reconstruction targets under stop-gradient. > **Q5: Asymmetric aggregation of time-series and image embeddings.** The asymmetry aggregation reflects their distinct scopes, since each temporal token represents a single modality (EEG, EOG, EMG, ECG, or respiration) whereas each image view already covers all modalities in the PSG window. Retaining separate temporal tokens preserves modality specific dynamics for flexible downstream weighting, while fusing CWT, STFT, and morphological views via learned weights provides a compact global summary without redundancy, jointly improving downstream performance. > **Minor issues: Mathematical notation and formatting.** Thank you for pointing this out. We will fix all the mathematical notation and formatting issues in the updated manuscript soon.I thank the authors for the additional experiments and clarifications. The new ablations help understand each component's contribution, and the explanations address several of my concerns regarding the implementation.
My main remaining concern is whether the benefits of using channel-semantic conditioning, one of the main contributions, appears marginal relative to the added complexity. Removing the channel-name encoder reduces accuracy by only 0.3 percentage points in Table 1. For context, changing the channel-naming condition in Table 3 produces a 0.2-percentage-point difference in accuracy.
I also appreciate the model-size comparison. Scaling is not the paper’s primary contribution, and proportional performance improvements are not expected. Nevertheless, the AUROC gain from Base to Large is small, and reporting the original metrics would make the comparison easier to interpret.
Overall, the rebuttal improves the clarity and supporting evidence, but it does not sufficiently resolve my concern about the demonstrated value of the paper’s central contribution. I therefore maintain my current rating.
Author Response (W1-W4)
Many thanks to Reviewer oBfh for providing detailed and in-depth reviews. We address your concerns by providing more experiments and clarifications as follows.
Table 1 presents ablations of channel semantics, image views, and pretraining objectives. Dropping any component lowers all three metrics, with inter modality reconstruction having the largest impact, confirming that all components complementarily contribute to the full model.
Table 1: Ablation study performance(%) for downstream sleep staging on the SHHS-1 dataset.
SemPSG computes reconstruction targets online using shared encoders, with stop-gradient applied to the target branch and no EMA teacher. Masked reconstruction is jointly optimized with cross-view and waveform-image InfoNCE losses from the start of training. These losses align representations of the same window and separate negatives, discouraging constant representations. We also use a 2,000-step learning-rate warmup, gradient clipping at norm 1, and a gradual increase in the cross-modal reconstruction weight to ease early optimization.
Pretraining completed full steps updates without reported NaNs. Final validation InfoNCE losses were 0.0130 (cross-view) and 0.00395 (waveform-image), supporting window-level discrimination and providing evidence against complete collapse at this level. Our assessment of training stability therefore rests on the joint objectives, optimization settings, and training records. We will add the figure of training and valiadation losses in the revised paper soon.
We have added Small/Large variants with 2.64/11.70M parameters. On downstream SHHS1 sleep staging, they achieve AUROC of 95.56% and 95.72%, respectively. The model in the original submission corresponds to the Base variant with 4.45M parameters, which achieves 95.69% AUROC. The figure below summarizes this comparison. We will add this comparison results to the revised paper soon.
Table 2: Architecture configurations and parameters across SemPSG scales.
To test whether the channel semantics tokenizer learns transferable physiological semantics rather than relying on familiar labels, we evaluated CFS while holding out all original channel names. We replaced all CFS channel names in all splits with equivalent physiological descriptions absent from the CNE pretraining corpus, such as changing the abbreviated label
THOR EFFORTtoThoracic Respiratory Effort, keeping signals and derivations fixed. Retraining the linear classifier on the frozen encoder with identical settings yields 82.1% Accuracy, 74.9% Kappa, and 73.7% Macro F1 , as shown in Table 3, matching the original familiar labels (81.9% Accuracy, 74.8% Kappa, and 73.6% Macro F1) and confirming transferable semantic understanding over label memorization.Table 3: Sleep staging performance (%) on CFS under original and unseen channel naming conventions.