acceptodds
Under review as a conference paper at ICLR 2027

Sleep2Word: Translating Peripheral to EEG using Physiological Wordization

Abstract

Estimating full-night electroencephalography (EEG) spectrograms from peripheral measurements could extend EEG-informed sleep analysis to recordings acquired without EEG. This task requires compact representations of signals with different temporal structures. We introduce Sleep2Word, a framework for translating respiration, electrocardiography (ECG), or photoplethysmography (PPG) into EEG spectrograms through duration-aware physiological words. Building on context-dependent chunking, Sleep2Word groups physiological embeddings into variable-duration chunks and jointly quantizes their content and duration into modality-specific wordbooks. A block-autoregressive translator then predicts target EEG word identities and occurrence-specific duration refinements within fixed-duration blocks. This design accommodates independent source and target wordbooks, limits temporal drift, and shortens the autoregressive decoding path. Trained on over 20K PSG cases across 9 datasets, Sleep2Word consistently outperforms existing spectrogram generation baselines in EEG spectrogram reconstruction, sleep staging, arousal detection, and survival prediction. Compressing full night sequences by up to 69.7%, the learned words yield a favorable accuracy–compute trade-off. Sleep event analysis further shows that the learned words capture physiologically plausible associations across modalities, suggesting their potential for interpretable cross-modal analysis.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Cross-modal translation, Sequence tokenization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mHXyOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies peripheral-to-EEG translation, bringing EEG-free recordings into EEG-informed analysis.

The authors adapt the dynamic chunking technique to physiological signals. Combined with duration-aware wordization, modality-specific discrete wordbooks are learned through self-supervised reconstruction. Each word encodes a variable-duration segment of physiological signal. Building on this latent representation, the authors train a peripheral-to-EEG translator. A chunk-based autoregressive generation scheme is introduced to mitigate the temporal drift issue.

Experiments show that the proposed method outperforms PasL and the C-T baseline in EEG translation quality. The generated EEG also achieves the best results among generation-based methods on downstream tasks, such as sleep staging, arousal detection, and survival prediction. Important component designs (such as chunking mechanism) are supported by ablation studies.

Critical strengths

Adapting dynamic chunking to physiological translation is non-trivial: since different modalities segment independently, source and target chunks have no positional correspondence, so naive solutions such as a transformer decoder or U-Net over chunks is not applicable. And once autoregressive generation is used to handle the target segmentation of unknown length, temporal drift could accumulate. The indroduced block-wise AR scheme is a reasonable answer to this adaptation problem.

Critical weaknesses

In abstract, the authors motivates the work as "extending EEG-informed sleep analysis to recordings acquired without EEG." However, the downstream evaluation actually performed involves "fine-tuning" classifiers on the generated EEG spectrograms, which supports a weaker claim: "the generated spectrograms contain task-relevant information." A fine-tuned classifier can adapt to systematic distortions in the generated spectrograms. Therefore, the experiments do not varify whether tools that cannot be "fine-tuned" (such as rule-based arousal detectors or black-box neural classifiers) would still work on generated EEG. A possible solution could be, to take a model pretrained on real EEG (ideally an external, frozen one), apply it zero-shot to the generated EEG, and compare across methods. Alternatively, the authors could tone down the abstract claim to match what the experiments actually show.

Questions for authors

In the original dynamic chunking, each chunk is represented directly by the latest embedding in the chunk, whereas Sleep2Word uses average pooling over the chunk. What motivated this change, and was any comparison between the two performed?

The generated spectrograms look smoother than the ground truth in Figure 8, could this limit its usage with existing EEG tools?

4 comments
AuthorsOpenReview ↗

Response to Reviewer mHXy (1/2)

We thank the reviewer for the insightful comments and for recognizing the challenge of adapting dynamic chunking to cross-modal physiological translation. We address each point below.

[W1] Evaluation with frozen classifiers trained on real EEG

We agree that fine-tuned downstream heads can adapt to systematic distortions in generated spectrograms. Following the reviewer's suggestion, we evaluated generated spectrograms with classifiers trained only on real EEG spectrograms and kept frozen, without any exposure to generated data. Specifically, we used the classifiers trained on the GT spectrograms of each training split (the GT columns of Tables 2 and 3) and applied them directly to the generated spectrograms of the corresponding test split.

Sleep staging (frozen GT-trained classifier)

DatasetMetricGT EEGRESP: OursRESP: PasLRESP: C-TECG: OursECG: C-TPPG: OursPPG: C-T
SHHS1MF10.8720.7100.6170.5730.6800.578––
Acc.0.8790.7050.5420.6160.6840.630––
Kappa0.8220.5850.4080.4200.5520.433––
MESAMF10.8580.7150.6750.5730.6420.4910.6420.208
Acc.0.8840.7380.5750.6450.6730.6190.6730.490
Kappa0.8250.6220.4900.4670.5280.3940.5270.033
CFSMF10.8620.6770.5880.5420.5690.4830.5540.099
Acc.0.8660.6750.4500.6120.5920.5710.6190.245
Kappa0.8060.5450.3410.4180.4200.3520.4260.000

Arousal detection (AUROC, frozen GT-trained classifier)

DatasetGT EEGRESP: OursRESP: PasLRESP: C-TECG: OursECG: C-TPPG: OursPPG: C-T
SHHS10.8790.8050.7870.7640.8130.748––
MESA0.8270.8200.7350.8100.7910.7190.7780.493
CFS0.8910.7940.7400.8010.7980.7490.6890.497

For sleep staging, Sleep2Word achieves the best performance among the generation methods in all settings. Compared with classifiers fine-tuned on generated spectrograms (Table 2), accuracy decreases by 0.045–0.094, indicating that part of the stage-relevant structure in the generated spectrograms differs from real EEG and is compensated for by fine-tuning. For arousal detection, Sleep2Word's AUROC stays within 0.017 of the fine-tuned setting (Table 3) in 7 of 8 settings. Sleep2Word achieves the highest AUROC among the generation methods except on CFS-RESP, where the C-T baseline is slightly higher (0.801 vs. 0.794). We conjecture that arousal detection performance is preserved because arousals are defined by abrupt, broadband shifts in EEG frequency lasting at least 3 seconds, which are well preserved at the coarse spectral level, whereas staging relies more on subtle and transient features that are partially smoothed in generation (see also our response to Q2).

External cohort: fully zero-shot pipeline.

To evaluate the setting where neither the generator nor the downstream classifier has seen the target cohort, we additionally used APPLES [1, 2], which is not used at any training stages of Sleep2Word and the baselines. Classifiers trained on GT spectrograms of SHHS1 and MESA were applied, without any adaptation, to spectrograms generated for all APPLES recordings (1097 recordings).

TaskMetricGT EEGRESP: OursRESP: PasLRESP: C-TECG: OursECG: C-T
Sleep stagingMF10.7380.5860.5200.5030.5300.481
Acc.0.8330.6560.6360.6250.6180.639
Kappa0.7250.4760.3910.3710.4040.356
Arousal detectionAUROC0.7810.7560.6370.7400.7210.665

Performance is lower than on the internal cohorts because the generation gap and the cohort shift compound. Nevertheless, Sleep2Word achieves the highest MF1 and kappa for both modalities; for ECG, the C-T baseline has slightly higher accuracy (0.618 vs. 0.639), but lower MF1 and kappa, which better reflect performance under class imbalance. For arousal detection, Sleep2Word from respiration reaches an AUROC of 0.756, within 0.025 of real EEG (0.781). We newly added “Evaluation with real EEG-trained classifier” paragraph under section 4.3 and added the results as Tables 5 and 6.

In our response to Reviewer 1LF2 and in Table 28 of the revised manuscript, we additionally report zero-shot age estimation results on APPLES. Sleep2Word also achieves the lowest estimation error compared to baselines for this task.

AuthorsOpenReview ↗

Response to Reviewer mHXy (2/2)

Scope of the claim. These results show that the generated spectrograms remain informative for classifiers trained only on real EEG, including on an unseen cohort, while a gap remains for sleep staging. We note that tools operating on raw EEG waveforms (e.g., rule-based detectors) cannot be applied directly, as Sleep2Word estimates spectrograms rather than waveforms. Following the reviewer's suggestion, we revised the abstract to state this scope precisely: “The generated spectrograms also remain informative for classifiers trained only on real EEG, including on an unseen cohort.”. We also added this as our limitation in Section 5.

[Q1] Average pooling vs. last-embedding chunk representation

We chose average pooling so that each word summarizes the whole segment it replaces. This is consistent with the rest of the pipeline: each word is scattered back to all positions of its chunk during dechunking. To verify this choice, we compared four strategies: average pooling, the first embedding, the last embedding, and attention pooling. We report respiratory-to-EEG sleep staging accuracy on SHHS1 under the linear-probing protocol (trained on SHHS1, as in Table 17 (previously Table 14)).

Chunk representationAcc.
Average pooling0.753
First embedding0.720
Last embedding0.735
Attention pooling0.714

Average pooling performs best, outperforming the last-embedding strategy by 0.018. We added this comparison to Appendix G and Table 22.

[Q2] Smoothness of generated spectrograms

We agree with this observation. Potential contributors include: (i) fine-grained EEG details that are not identifiable from peripheral signals tend to be estimated as their conditional average under the reconstruction objective, (ii) each word is a discrete prototype shared across many segments, and (iii) the MSE objective favors smooth outputs. The zero-shot results in our response to W1 indirectly quantify the practical impact: features at the coarse spectral level, such as arousal-related shifts, transfer well to classifiers trained on real EEG, whereas sleep staging shows a measurable gap. We added this as a limitation in Section 5, noting that the generated spectrograms support utility for the evaluated spectrogram-based tasks, while fine-grained temporal fidelity remains unverified. Reducing over-smoothing, for example with band-specific reconstruction losses or generative objectives that sample from the conditional distribution, is a promising direction for future work.

We thank the reviewer again for the constructive suggestions, which have helped us state the scope of our claims more precisely.

References

[1] Quan, S. F., et al. "The association between obstructive sleep apnea and neurocognitive performance—the Apnea Positive Pressure Long-term Efficacy Study (APPLES)." Sleep 34.3 (2011): 303–314.

[2] Zhang, G.-Q., et al. "The National Sleep Research Resource: towards a sleep data commons." JAMIA 25.10 (2018): 1351–1358.

I appreciate the author’s additional experiments, which fully address my concerns; I have therefore increased the Soundness rating. I view this work as a solid engineering adaptation of existing techniques, and my final rating remains unchanged.

We sincerely thank the reviewer for the thoughtful engagement and for raising the Soundness rating. We are glad that the additional experiments fully addressed the reviewer's concerns. Thank you again for the constructive feedback, which has substantially improved the paper.

Reviewer_cGETOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Sleep2Word estimates full-night EEG spectrograms from respiration, ECG, or PPG. It combines a modified SleepMaMi encoder with variable-duration chunking, joint content-duration quantization, and autoregressive decoding within fixed time blocks. The experiments cover reconstruction, sleep staging, arousal detection, survival prediction, and transfer of the frozen generator to CFS. The paper reports improvements over the tested spectrogram-generation baselines, shorter token sequences, and a favorable accuracy-compute trade-off.

Critical strengths
  1. The method addresses the timing mismatch that arises when source and target modalities are segmented independently. Predicting word durations within shared recording intervals makes sense for this problem (pp. 5-6). Table 15 provides some support: staging accuracy rises from 0.732 with content-only words to 0.748 with duration-aware words and 0.753 with duration residuals. The paper also acknowledges the methods on which these components build.

  2. The evaluation covers several peripheral modalities and downstream tasks. Table 1 reports consistently better reconstruction than the tested generation baselines, and Tables 2-3 report gains across cohorts. The authors distinguish estimated spectral structure from recovery of the exact EEG waveform (p. 1). They also state that CFS uses locally trained downstream heads after transfer of a frozen generator (p. 15), making the scope of that experiment clear.

Critical weaknesses
  1. Table 14 does not establish that adaptive and fixed pooling were compared at the same realized compression rate (pp. 20-21). FixedPool averages four base embeddings and retains about 25% of the base tokens. Table 19 separately reports retention of 35.1% for respiration and 32.4% for EEG. These measurements come from different checkpoints than the SHHS1-only ablation, so I cannot conclude that Table 14 used unequal rates. Still, nominal N=4 alone does not establish a matched comparison. Without the ablation's actual token counts, the 0.020 accuracy gain could partly come from retaining more tokens. Please report the actual source and target token counts for each variant. A nearby fixed-pooling configuration with matched lengths would help resolve this within the existing experiment.

  2. Figure 3 does not define what its cumulative training TFLOPs include (p. 8). The pipeline involves modified SleepMaMi pretraining, chunking, wordization, translation, and downstream evaluation (pp. 16-19). I could not tell which stages and forward passes through frozen modules were counted, or whether preprocessing and cached embeddings were handled consistently across methods. Please give the compute breakdown and counting convention for each method. Reporting translation-only or amortized costs would also be useful, provided they are labeled clearly. The curve favors Sleep2Word under the reported accounting, but I cannot infer the cost of the complete pipeline from it.

Questions for authors
  1. What are the actual source and target token counts for the Table 14 variants, and does the gain remain when those counts are matched? If the gain persists, I would be more confident that adaptive segmentation accounts for the improvement.

  2. Which training stages and frozen forward passes are included in Figure 3? Please give a consistent breakdown across methods so that I can judge whether the efficiency advantage holds under that accounting convention.

4 comments
AuthorsOpenReview ↗

Response to Reviewer cGET (1/2)

We thank the reviewer for the careful reading and the specific, actionable suggestions. Below we address each point; all new results are included in the revised manuscript.

[W1, Q1] FixedPool vs. Sleep2Word at approximately matched token counts

We agree that the nominal ratio N=4 alone does not establish a matched comparison. We measured the realized token counts using the SHHS1-only ablation checkpoints used in the original Table 14 (Table 17 in the revision), averaged over 50 SHHS1 test recordings. We also trained FixedPool with a pooling window of 3 (FixedPool-3), whose token count is closest to that of Sleep2Word.

MethodSource (RESP) tokensTarget (EEG) tokensRetained (Src / Tgt)Acc.
Base2base27,93827,938100% / 100%0.731
FixedPool-46,9856,98525.0% / 25.0%0.733
FixedPool-39,3139,31333.3% / 33.3%0.736
Sleep2Word9,0809,28032.5% / 33.2%0.753

Relative to FixedPool-3, Sleep2Word retains 0.8 percentage points fewer source tokens and 0.1 percentage points fewer target tokens. Under this near-matched setting, adding tokens to FixedPool improves accuracy by only 0.003, while Sleep2Word remains 0.017 higher. Two further observations indicate that token count does not explain the gain: (i) Base2base, which retains all tokens, underperforms both pooling variants and Sleep2Word; and (ii) in Table 20, lowering the target chunking ratio from N=4 to N=2 (roughly doubling the number of tokens) leaves Sleep2Word's accuracy unchanged (0.757 in both cases). We therefore attribute the improvement mainly to adaptive boundary placement. We added the token counts and the FixedPool-3 result to Table 17 (previously Table 14).

AuthorsOpenReview ↗

Response to Reviewer cGET (2/2)

[W2, Q2] Accounting convention and compute breakdown for Figure 3

We thank the reviewer for pointing out this ambiguity. Figure 3 counts the training compute of each method's translation pipeline under the following convention:

  • Included: all training stages of each method. For Sleep2Word, we include chunking and wordization training for both modalities, as well as peripheral-to-EEG translator training.
  • Frozen modules: SleepMaMi features are cached during Sleep2Word training. During wordization stage, chunking/dechunking modules are frozen. During translation stage, chunking/dechunking and wordization modules are frozen. We included FLOPs of the forward pass for any modules that are frozen.
  • Excluded: the one-time pretraining of the frozen feature extractors (SleepMaMi for Sleep2Word and Base2base; VQ-GAN for PasL), SleepMaMi cache creation and the downstream evaluation.

Because the excluded pretraining and caching costs differ across methods, we also report totals that include them. The table below summarizes TFLOPs used for each method.

MethodEpochsPer-epochTotal (Fig. 3)Feature-extractor pretraining / CachingTotal incl. Pretraining and cachingAcc.
Sleep2Word377,019231,056278,126 / 35,664 (SleepMaMi)544,8460.743
5385,093698,8830.751
10770,1861,083,9760.756
Base2base1195,664195,664278,126 / 35,664 (SleepMaMi)509,4540.667
3586,993900,7830.721
5978,3221,292,1120.731
C-T baseline1041,403414,035–414,0350.531
20828,070828,0700.571
301,242,1051,242,1050.621
PasL407,029281,1472,756,452 / 0 (VQGAN)3,037,5990.625
75527,1513,283,6030.635
5003,514,3436,270,7950.641

The table below provides per-epoch breakdown of Sleep2Word training compute by stage

StageTFLOPs per epoch
Chunking34,449
Wordization11,992
Translation30,578
Total77,019

Under both accounting conventions, Sleep2Word retains a favorable training accuracy–compute trade-off across the evaluated configurations. Including pretraining, Sleep2Word reaches 0.743 at 0.545M TFLOPs, which already exceeds the best accuracy of every baseline at any tested budget.

For completeness, we also report inference cost per night.

MethodInference TFLOPs (per night)
Sleep2Word1.8
Base2base3.5
C-T baseline0.7
PasL0.3

Owing to the shorter sequences, Sleep2Word halves the inference cost of Base2base. It is higher than that of C-T and PasL, mainly due to the SleepMaMi encoder and autoregressive decoding. In absolute terms, this corresponds to 0.23 seconds per night on a single H200 GPU. Per-night inference latency was measured with a batch size of one recording, using 10 SHHS1 test nights (921 average number of epochs; internally padded to a max_epochs=1200). The timed path is end-to-end from the raw source signal to the EEG spectrogram.

To avoid overstating our claim, we revised "superior computational efficiency" in the Introduction to refer specifically to the training accuracy–compute trade-off. We added the counting convention and all tables above to the newly added section “Compute Efficiency Details” in Appendix and Table 29, 30 and 31

We hope these additions resolve the reviewer's concerns, and we would be happy to provide further details.

Reviewer_cGETOpenReview ↗

Reviewer final reply: Upgrade from 2 to 3

The reason I haven’t responded until now is that I took the time to review the revised PDF. The authors have highlighted all changes in blue and expanded the manuscript to 10 pages—this is a positive improvement. I have also reviewed the feedback from other reviewers. I recognize the significant effort put into this work. I intend to raise my score to 3, but I would like to clarify my reasoning. To the AC: Please note that while I am increasing my score to 3, it is a "weak 3"—leaning closer to 2.5. Why am I leaning toward a 3? While the research significance of this paper is not particularly striking, its methodological contributions and the completeness of the experiments—especially following the rebuttal—reach a level that warrants consideration for a "weak accept."

  1. The methodological innovation is not groundbreaking, yet it goes beyond a simple assembly of existing techniques. Since the "PasL" approach (from ICML 2026) has already explored generating EEG time-frequency spectra from respiratory signals, "Sleep2Word" cannot be considered a pioneering work. However, the paper does address specific technical challenges—such as handling correspondences between tokens of unequal lengths, managing generation duration, and accounting for temporal drift when dealing with independently segmented modalities. The "Duration-aware wordization" and "Block-AR" mechanisms are well-targeted design choices.
  2. The authors’ rebuttal contains substantive content.
  3. The primary issue with the paper lies in its significance, not its soundness. Why is it also very close to a 2? I largely agree with the core concern raised by Reviewer 1LF2: If sleep analysis can be performed directly using peripheral physiological signals, why is it necessary to generate an intermediate representation—an EEG spectrogram—that is of lower quality than the actual EEG data?

Furthermore, looking at the additional experiments provided by the authors on the APPLES dataset:

  • Performance on sleep staging still lags behind "wav2sleep."
  • Performance on arousal detection is roughly on par with "wav2sleep."
  • The improvements gained from feature fusion are quite limited.
  • The generated EEG spectrograms still exhibit significant loss of detail. In short, while the paper demonstrates that "we are better at generating EEG spectrograms," it fails to sufficiently prove that "generating EEG spectrograms is a task truly worth pursuing."

While this does not constitute a fatal technical flaw, it does represent a significant concern regarding the research value of the work, especially for a paper at the ICLR level.

We sincerely thank the reviewer for carefully reviewing the revised manuscript and the discussions with other reviewers, and for raising the score. We appreciate the balanced assessment and agree that Sleep2Word builds on PasL, which first established the feasibility of respiration-to-EEG synthesis; our contribution lies in addressing the technical challenges that arise with different modalities.

Regarding the remaining concern on significance, we agree with the reviewer's reading of the APPLES results: for sleep staging, a specialized model trained directly on peripheral signals performs better, and the fusion gains are modest. We will state explicitly in the paper that, for a single task with abundant labels, direct peripheral modeling is preferable. We see the value of generated spectrograms elsewhere, namely in analyses that rely on EEG-domain models or measures that are unavailable for peripheral signals:

  • Reusing EEG-domain models. An age regressor trained only on real EEG, applied to Sleep2Word outputs, achieves lower error on the unseen APPLES cohort than a regressor trained directly on wav2sleep features (10.36 vs. 10.79 years). Classifiers trained only on real EEG also remain effective on generated spectrograms (See response to mHXy). Direct peripheral modeling cannot use such EEG-domain models without collecting new labels in the peripheral domain, which matters for large peripheral-only cohorts such as home sleep apnea tests and wearables.
  • Interpretable EEG-domain outputs. Sleep2Word outputs an EEG spectrogram that clinicians and researchers can read and analyze with standard spectral measures. It also enables model-based exploration of cross-modal physiology, such as the increase of arousal-related EEG words after apnea termination (Figure 4c–d).
  • EEG spectral measures beyond stage labels. Generated spectrograms provide continuous spectral measures that cannot be obtained from stage predictions.

New result: spectral slow-wave activity (SWA). To substantiate the second point, we analyzed SWA (normalized 0.5–4 Hz power) computed from Sleep2Word outputs (RESP-to-EEG). Continuous SWA carries clinically relevant information beyond stage-based measures: lower NREM SWA, particularly at 1–2 Hz, is associated with tau pathology in early Alzheimer's disease [1], and reduced SWA mediates age-related memory impairment [2].

MetricDatasetReal EEGSleep2WordC-T baseline
Pearson corr. with real-EEG SWA (whole night)SHHS1–0.7470.567
APPLES–0.6270.503
Pearson corr. with real-EEG SWA (N2+N3)SHHS1–0.7240.431
APPLES–0.6970.414
ΔSWA (first − second half of night), medianSHHS10.03760.04300.0030
APPLES0.04150.05220.0034
Spearman corr. between age and ΔSWASHHS1−0.184−0.1530.026
APPLES−0.069−0.0210.004

Night-level SWA from Sleep2Word correlates closely with real-EEG SWA on both SHHS1 and APPLES, substantially more than the C-T baseline. Sleep2Word also reproduces the decline of SWA across the night that reflects sleep homeostasis, whereas the C-T baseline shows an almost flat profile, and on SHHS1 it preserves the negative association between age and this decline (−0.153 vs. −0.184 in real EEG). This also relates to the reviewer's point on the loss of detail: although fine-grained details are smoothed in generated spectrograms, broadband spectral dynamics such as SWA and its within-night trajectory are preserved. We added this analysis to Appendix L and full-night SWA trajectories as Figure 9 in the revised manuscript.

We hope these results clarify when generating EEG spectrograms is worth pursuing, and we thank the reviewer again for the thorough and constructive engagement throughout the discussion.

[1] Lucey, B. P., et al. "Reduced non–rapid eye movement sleep is associated with tau pathology in early Alzheimer's disease." Science Translational Medicine 11.474 (2019): eaau6550. [2] Mander, B. A., et al. "Prefrontal atrophy, disrupted NREM slow waves and impaired hippocampal-dependent memory in aging." Nature Neuroscience 16.3 (2013): 357–364.

Reviewer_1LF2OpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The motivation behind this work is to enable EEG-informed sleep analysis without needing real EEG signals. EEG is generally hard to collect outside sleep laboratories. Thus, the work presents a method for generating EEG spectrograms from peripheral signals such as ECG, respiratory signals, and PPG.

The method incorporates a "wordbook" and "wordization" mechanism, which is similar to a codebook. The work claims to compress up to 69.7% of the original signals into learned words. With the learned words, the model then generates the EEG spectrogram, which enables downstream tasks such as sleep analysis. Some technical contributions include variable-length chunking for compute efficiency.

The work evaluates its model on 2 internal and 1 external datasets, where they achieve the best performance on reconstruction and downstream sleep analysis tasks.

Critical strengths
  1. The paper addresses a relevant problem: whether peripheral physiological signals can be used to estimate EEG spectrograms during sleep.

  2. Sleep2Word achieves the best reconstruction performance among the compared spectrogram-generation methods across all reported dataset and modality combinations. The evaluation also on an external setting CFS. Further evaluates the generated spectrograms on multiple downstream tasks, including sleep staging, arousal detection, and survival prediction.

  3. The authors propose a variable chunking method to increase compute efficiency.

Critical weaknesses

Methodology

  1. The motivation for representing physiological signals as "words" is not convincing. The model should be able to perform the same operations and achieve similar performance when representing the chunks as vector embeddings in codebooks.

  2. For sleep staging, the generated EEG spectrogram shows worse performance than both the GT EEG spectrogram and wav2sleep. From this, we can conclude that the generated EEG spectrogram from peripheral signals alone is not better than the original peripheral signals. However, I am curious whether training a model to predict sleep stages using both the original peripheral signals (ECG, respiration, or PPG) and the generated EEG spectrogram would lead to better performance than wav2sleep or even the GT EEG spectrogram.

Results & Experimental Setting

  1. The baseline choices are limited. They include only one EEG generation model and a naive CNN-Transformer model. I suggest incorporating some diffusion-based reconstruction or translation methods, such as RDDM [1] or even adapt CSDI [2]

  2. In Table 2, all entries perform worse than the specialized model wav2sleep. This undermines the utility and need for EEG spectrograms in sleep staging. If respiratory signals, ECG, or PPG alone can already provide better sleep staging accuracy, why do we need to generate EEG spectrograms to perform sleep staging? It seems to be meaningless to add an intermediate step where errors can accumulate.

  3. The work does not include any external validation for downstream sleep analysis tasks. It only includes an external setting for reconstruction but lacks downstream analysis. This undermines the model's performance and generalization under external settings. In particular, reconstruction performance is already generally worse in the external setting than in the internal setting. These reconstruction errors may propagate to downstream tasks and negatively affect their performance.

Minor

  1. Why use a "micro-encoder"? A comparison with a sleep foundation model-based encoder would help justify whether a micro-encoder is sufficient.

  2. The experimental data setting is confusing. The training datasets in Table 5 are listed as 1. PHY, 2. SHHS1, 3. SHHS2, 4. KISS, 5. KVSS, 6. MrOS1, 7. MrOS2, and 8. MESA, but the abstract mentions that the model was trained on 9 datasets.

[1] https://ojs.aaai.org/index.php/AAAI/article/view/29422/30683 [2] https://proceedings.neurips.cc/paper_files/paper/2021/file/cfe8504bda37b575c70ee1a8276f3486-Paper.pdf

Questions for authors

Please refer to the mentioned weakness. My major questions are as below but are not limited to these.

  1. For sleep staging, if wav2sleep already performs better without having to generate EEG spectrograms, what is the practical benefit of generating it?

  2. How well does the complete pipeline (including the downstream tasks) generalize to an external setting without fine-tuning?

  3. The benefits of discrete "words" are not convincing. A simple VQ-VAE or VAE approach may do its job.

9 comments
AuthorsOpenReview ↗

Response to Reviewer 1LF2 (1/5)

We thank the reviewer for the detailed and constructive feedback. Following the suggestions, we added diffusion baselines, a fusion study with wav2sleep, a plain VQ baseline, and an evaluation on an additional external cohort. We address each point below.

[W-M1, Q3] Benefit of duration-aware words over plain vector quantization

We agree that our wordbook is a vector-quantized codebook. The difference from plain VQ lies in what is quantized: (i) variable-duration chunks whose boundaries are learned from context, rather than fixed-size patches, and (ii) a joint content–duration space, so that each word encodes both what occurs and how long it lasts. To isolate these factors, we compare the variants below using respiratory-to-EEG sleep staging accuracy on SHHS1 (linear probing, trained on SHHS1). Base2base + VQ quantizes the source and target base embeddings with a VQ-VAE codebook (K = 2048, matching the wordbook size). Chunk2chunk employs dynamic chunking [1], but translates continuous source chunks into continuous target chunks and their boundaries without wordbooks.

VariantSegmentationDiscrete codesDuration in codeAcc.
Base2baseFixed (1 s)✗–0.731
Base2base + VQ (new)Fixed (1 s)✓–0.716
FixedPoolFixed (4 s)✗–0.733
Chunk2chunk (new)Adaptive✗–0.637
Content-only wordbookAdaptive✓✗0.732
Sleep2WordAdaptive✓✓0.753

The table above summarizes the results. First, plain VQ at a fixed resolution degrades performance (0.731 → 0.716), so discretization alone is not beneficial. Second, adaptive chunks without discretization perform poorly (0.637). The original dynamic chunking reuses the source boundaries for dechunking, which is valid when source and target share the same segmentation, as in language modeling, but not across modalities with heterogeneous boundaries. Predicting continuous target chunks together with their boundaries is difficult, whereas the wordbook turns translation into classification over reusable prototypes. Third, adding duration to the code and occurrence-specific residuals add 0.021. These results support our particular duration-aware formulation rather than discretization alone.

Compression also makes full-night context affordable. Prior works have shown that full-night context sleep modeling [2–4] improves the model performance. To confirm this, we compared encoders with a 3-epoch context and a full-night context (Train and test on SHHS1, linear probe).

ContextPFLOPsTrain MAE ↓Test MAE ↓Acc. ↑
3 epochs300.00390.04480.663
900.00380.04430.674
3020.00340.04500.669
Full night320.00470.04380.702
960.00380.04310.726
3180.00350.04220.731

With a 3-epoch context, test accuracy saturates and then declines while train MAE keeps decreasing, whereas the full-night model keeps improving. Because attention over an uncompressed full-night sequence is costly, Sleep2Word shortens the sequence by 64–70% (Table 23 (previously Table 19) ), achieving a better accuracy–compute trade-off (Figure 3; see also our response to Reviewer cGET). Finally, words provide an interpretable unit of analysis as presented in our qualitative analysis (Figure 4c–d).

AuthorsOpenReview ↗

Response to Reviewer 1LF2 (2/5)

[W-M2, W-R2, Q1] Practical benefit of generating EEG spectrograms

The generated spectrogram is designed as a task-agnostic intermediate rather than a replacement for task-specific models. Although Sleep2Word underperforms wav2sleep in sleep staging, for which the entire model of wav2sleep is directly supervised, it outperforms wav2sleep in arousal detection on all datasets and modalities (Table 3) and achieves higher or comparable C-indices in survival prediction (Table 4). We further examined its benefit with two new experiments.

(1) Complementarity to peripheral signals. Following the reviewer's suggestion, we tested whether generated spectrograms provide information complementary to the peripheral signals. We concatenated the 128-d features of the frozen pretrained wav2sleep backbone with the 128-d features of a frozen CNN classifier trained for sleep staging on Sleep2Word spectrograms, and trained a linear classifier on the 256-d features. Heads were trained on each dataset's training split and evaluated on the test split.

DatasetSourcewav2sleep MF1Acc.Kappa+ Sleep2Word MF1Acc.Kappa
SHHS1RESP0.7740.7980.6900.7860.8060.705
SHHS1ECG0.7730.7960.6890.7870.8090.708
MESARESP0.7530.8330.7340.7590.8360.739
MESAECG0.7310.8050.6910.7520.8220.717
MESAPPG0.7220.8070.6930.7350.8150.706
CFSRESP0.7990.8100.7190.8110.8220.735
CFSECG0.8270.8360.7570.8270.8380.759
CFSPPG0.7800.7940.6940.7830.7970.697

(As noted in our global response, we corrected the wav2sleep results to ensure consistent evaluation windows across methods. In the original manuscript, Sleep2Word and other baselines were evaluated after excluding wake epochs occurring more than 30 minutes before sleep onset or more than 30 minutes after the end of sleep, whereas wav2sleep was evaluated on untrimmed recordings. We have now recomputed wav2sleep’s metrics using the same epoch-selection criteria as Sleep2Word. This correction lowers the reported wav2sleep scores but leaves the relative ranking of the methods unchanged. We apologize for this inconsistency and have highlighted the corrected results in blue in the revised manuscript.)

Adding generated-spectrogram features improves accuracy and Kappa in all eight dataset–modality combinations and MF1 in seven (tied for CFS-ECG). This indicates that the generated spectrograms carry information not fully captured by a representation specialized for sleep staging. We added this result in Section 4.3 and Table 2.

(2) A new downstream task: Age estimation. Age estimation from sleep EEG is an established task, and brain age index (the gap between predicted and chronological age) is associated with dementia [5], mortality [6], and neuropsychiatric disease [7]. We trained an age regressor on GT EEG spectrograms from SHHS1, MESA, and CFS and applied it, without adaptation, to spectrograms generated from RESP for APPLES [8, 9], a cohort unseen by all models. For wav2sleep, a regressor was trained on its features from the same training cohorts.

InputMAE in years (↓)
GT EEG8.74
Sleep2Word10.36
C-T baseline13.72
wav2sleep features10.79

Sleep2Word yields lower error than wav2sleep features, even though the regressor was never exposed to generated spectrograms. This supports the utility of generated spectrograms as inputs to a frozen EEG-trained age regressor. We added this result in Section J.2.

AuthorsOpenReview ↗

Response to Reviewer 1LF2 (3/5)

[W-R1] Additional baselines

As proposed by the reviewer, we implemented DDPM [10] and RDDM [11]. Since RDDM was designed for PPG-to-ECG waveform translation and set its regions of interest as QRS region, it cannot be readily applicable for peripheral-to-EEG tasks. To tackle this, we set its regions of interest to the delta (0.5–4 Hz) and alpha–sigma (8–16 Hz) bands of the EEG spectrogram. Training and sampling details are added in Appendix C. CSDI is designed for imputing missing values within a time series; as DDPM and RDDM already represent conditional diffusion for physiological translation, we prioritized them within the rebuttal period.

Reconstruction

DatasetMetricRESP DDPMRESP RDDMRESP OursECG DDPMECG RDDMECG OursPPG DDPMPPG RDDMPPG Ours
SHHS1MAE (↓)0.0950.0770.0470.0940.0710.048–––
SNR (↑)11.3512.8817.3511.4013.7017.26–––
MESAMAE (↓)0.1180.0860.0560.1210.0840.0590.1190.0910.057
SNR (↑)8.8910.7814.388.7311.0614.128.7910.3014.23
CFSMAE (↓)0.1130.0790.0580.1220.0800.0650.1210.0900.059
SNR (↑)9.2911.8215.268.7611.8214.388.7710.5614.69

Sleep staging

DatasetMetricRESP DDPMRESP RDDMRESP OursECG DDPMECG RDDMECG OursPPG DDPMPPG RDDMPPG Ours
SHHS1MF10.5590.5450.7570.5180.5520.714–––
Acc.0.5870.5820.7660.5720.5750.729–––
Kappa0.3930.3730.6580.3460.3850.603–––
MESAMF10.5860.5500.7570.5110.5030.6990.4390.4720.709
Acc.0.6610.6090.7990.5760.5840.7410.5480.5650.754
Kappa0.4750.3990.6940.3660.3570.6090.2900.3310.626
CFSMF10.5210.5060.7430.4650.4930.6700.3740.3470.679
Acc.0.5530.5510.7480.5170.5230.6860.5100.5410.688
Kappa0.3570.3340.6390.2770.3100.5460.2080.2260.550

Arousal detection (AUROC)

DatasetRESP DDPMRESP RDDMRESP OursECG DDPMECG RDDMECG OursPPG DDPMPPG RDDMPPG Ours
SHHS10.6340.6450.7990.6290.6600.802–––
MESA0.6410.6530.8240.6140.6320.7790.4900.5010.781
CFS0.6320.6580.7770.6380.6590.7880.5530.5220.757

Survival prediction on SHHS1 (C-index)

OutcomeRESP DDPMRESP RDDMRESP OursECG DDPMECG RDDMECG Ours
All-cause mortality0.5700.5870.7000.7170.6360.736
CVD mortality0.5620.5760.8320.7590.7210.826
Stroke0.5240.5300.8240.8020.5590.837

Sleep2Word outperforms both diffusion baselines across all datasets, modalities, and tasks. Although RDDM improves reconstruction over DDPM, it does not consistently improve downstream performance. We added these baselines to Tables 1–4.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global response to reviewers

We sincerely thank all reviewers for their careful reading and constructive feedback.

The reviewers recognized our strengths:

  • Consistent improvements over spectrogram-generation baselines in reconstruction and downstream tasks (1LF2, cGET, mHXY)
  • Relevance of the peripheral-to-EEG translation problem (1LF2)
  • Duration-aware design and Block-AR decoding that address the timing mismatch between independently segmented modalities (cGET, mHXY)
  • Broad evaluation across peripheral modalities and downstream tasks, including the held-out CFS cohort (1LF2, cGET)
  • Ablations supporting the core design choices (mHXY)
  • Improved compute efficiency through variable-length chunking (1LF2)

Main concerns

The reviewers requested stronger evidence for:

  • The benefit of duration-aware wordization over standard vector quantization and continuous representations (1LF2).
  • The practical value of generated spectrograms relative to direct peripheral-signal models such as wav2sleep (1LF2).
  • External generalization of the complete prediction pipeline without adaptation (1LF2).
  • Compatibility with frozen classifiers trained only on real EEG (mHXY).
  • Comparisons with diffusion-based generation methods (1LF2).
  • Adaptive chunking at comparable realized token counts and transparent compute accounting (cGET).
  • The choice of average pooling and the implications of spectrogram smoothing (mHXY).
  • clarification of the encoder choice and the number of training datasets (1LF2)

We address these points in the individual responses.

Summary of revisions

The revised manuscript includes the following additions and clarifications:

  • Additional generation baselines. We added conditional DDPM and adapted RDDM comparisons for reconstruction, sleep staging, arousal detection, and survival prediction, together with their training and sampling settings (Tables 1–4 and 24–25; Appendix C, Table 15).
  • Feature fusion and an additional downstream task. We evaluated the combination of wav2sleep features and features derived from Sleep2Word-generated spectrograms (Tables 2 and 24). We also added age estimation on APPLES using a frozen regressor trained on real EEG spectrograms (Appendix J.2, Table 28).
  • External evaluation without adaptation. We added APPLES as a cohort unseen during model training and evaluated reconstruction and downstream predictions using models trained on other cohorts (Tables 1–3; Appendix A.8).
  • Frozen real-EEG-trained classifiers. We directly applied classifiers trained only on real EEG to generated spectrograms, including on APPLES, without adaptation to generated data (Tables 5–6 and 26–27). We revised the abstract and added limitations to clarify the scope of these findings.
  • Expanded representation ablations. We added fixed-resolution VQ and continuous-chunk baselines, reported realized source and target token counts, and included FixedPool-3 for comparison at approximately matched sequence lengths (Appendix G, Table 17).
  • Chunk representation comparison. We compared average pooling with first-embedding, last-embedding, and attention-pooling representations (Appendix G, Table 22).
  • Additional compute reporting. We added training-cost totals including feature-extractor pretraining and caching, together with inference FLOPs and latency measurement conditions (Appendix K, Tables 29–31). We narrowed the efficiency claim to the training accuracy–compute trade-off.
  • Dataset-count correction. We corrected the training corpus description from nine to eight datasets in the abstract and Section 4.1. Changes are highlighted in blue in the revised manuscript.

Correction to the wav2sleep evaluation

Separately, we identified and corrected an inconsistency in the evaluation windows used for wav2sleep. In the original manuscript, Sleep2Word and other baselines were evaluated after excluding wake epochs occurring more than 30 minutes before sleep onset or more than 30 minutes after the end of sleep, whereas wav2sleep was evaluated on untrimmed recordings. We recomputed the wav2sleep results using the same epoch-selection criteria as Sleep2Word. The corrected sleep-staging scores are lower, but relative ranking is maintained. We apologize for this inconsistency; the corrected values are highlighted in blue.

We thank the reviewers again for helping us strengthen the evaluation and clarify the scope of the paper’s claims.

AuthorsOpenReview ↗

Summary of discussion and revision

We thank all reviewers for the constructive and engaged discussion.

Main concern and how we addressed it.

Following the rebuttal, Reviewer mHXY noted that the additional experiments addressed their concerns, and Reviewers cGET and 1LF2 acknowledged the strengthened evidence from the new baselines and the external evaluation. The remaining concern, shared by Reviewers cGET and 1LF2, was the significance of generating EEG spectrograms when peripheral signals can be modeled directly. Reviewer 1LF2 found our use cases, reusing EEG-domain models and interpreting EEG-domain outputs, convincing and suggested restructuring the manuscript around them. We have revised the manuscript accordingly (changes in blue).

Key additions during the discussion

  • External evaluation on APPLES, including a fully zero-shot pipeline.
  • Diffusion-based baselines (DDPM, RDDM), which Sleep2Word outperforms in reconstruction and all downstream tasks.
  • Reuse of classifiers trained only on real EEG directly applied to generated spectrograms.
  • Slow-wave activity (SWA) analysis: generated SWA correlates closely with real-EEG SWA on SHHS1 and APPLES.
  • Age estimation with an EEG-trained regressor, and fusion with wav2sleep showing complementary information.
  • Token-count-matched chunking ablation and a stage-wise compute breakdown.

Manuscript revision in current update

  • Introduction: enabling EEG-domain models and analyses for recordings without EEG is stated as a primary motivation.
  • Experiments are reorganized as follows:
    • 4.1 Datasets, Preprocessing, and Baselines
    • 4.2 Spectrogram Reconstruction Performance
    • 4.3 Reusing EEG-Domain Models and Interpreting Outputs (real-EEG-trained classifiers, SWA analysis, apnea–arousal coupling)
    • 4.4 Downstream Performance (task-specific heads and fusion with wav2sleep)
    • 4.5 Compute Efficiency and Ablation
  • A limitations paragraph discussing the smoothness of generated spectrograms and the scope of our claims.

Due to the limited time, we could not include the following, but we commit to additionally include:

  • DDPM and RDDM results in the real-EEG-trained classifier experiments (Section 4.3).
  • SWA analysis for ECG and PPG inputs and for the MESA and CFS cohorts.

We thank the reviewers again for their thoughtful participation, which has substantially improved the focus and clarity of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.