acceptodds
Under review as a conference paper at ICLR 2027

wHuBERT: Layer Selection for Iterative Masked-Prediction Pre-training on Low-Resolution Wearable Biosignals

Abstract

Applications for wearable biosignals increasingly use the foundation model paradigm, where models are pre-trained without labels. HuBERT, originally developed for speech, requires two pre-training phases, the first one with coarse clustered pseudo-labels, and the second one with clusters derived from the representations of an intermediate layer from the first phase. The selection of the correct layer for the second iteration is crucial for a well-performing model. However, accurate performance metrics can only be obtained after the two pre-training phases and one fine-tuning phase for a downstream task. This represents a large commitment of compute before any concrete metric is available. One clustering layer also forces a trade-off between tasks and undercuts the "train once, apply anywhere" premise. We measure the layer selection effect on wHuBERT, a Conformer encoder pre-trained on photoplethysmography and accelerometry sampled at 25 Hz, the rate consumer wearables use. We propose a cheap probe-based protocol, applied to every layer at regular checkpoints, that selects the best-scoring layer for the given task. Across 2 independent runs, the final-checkpoint best-scoring layer for sleep staging lies just below the top, rarely at the final layer. We also computed 7 label-free statistics, including estimates of effective rank, but show they are not reliable for layer selection. We verify these results on two public sleep-stages corpora. We also fine-tune and distill an iteration-2 encoder and both the teacher and student score above the baseline from a consumer device. The protocol thus offers low-cost layer selection for foundation models on low-resolution biosignals.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, masked prediction, HuBERT, layer selection, representation quality, RankMe, effective rank, wearable biosignals, photoplethysmography, sleep staging

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_Ux3QOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes wHUBERT, a Conformer encoder adapted from HUBERT pretrained on wearable sensor signals for sleep staging. The main contribution is a protocol for selecting which iteration-1 layer of wHUBERT to use for clustering in iteration-2, by freezing iteration-1 checkpoints and fitting a probe on each of the candidate layers. This paper also tests whether label-free statistics can be used instead of this protocol, where the results indicate otherwise.

As the paper currently stands, it does not appear to be mature enough for publication, mostly due to the narrow scope of its contributions and lacking supporting evidence for its claims.

Critical strengths
  1. The authors are candid about the limitations of their work.
  2. The untrained-encoder control adds meaningful information as it shows that the interior peak in the probe comes from pretraining rather than architecture alone.
Critical weaknesses
  1. The authors did not test whether the layer choice matters. To validate this protocol, the authors should train iteration-2 encoders from layers other than 10, but that was not done. Hence there is no evidence that this protocol provides meaningful selection at all. This will be the single experiment that will go a long way to strengthen this paper. The authors also claim that the protocol is a computationally cheaper alternative, but provide no evidence at all to support it.
  2. The authors motivate this work by saying that "One clustering layer also forces a trade-off between tasks", suggesting that different tasks benefit from choosing different clustering layers. But the paper only tests one task. Uncited precendents also severely undercut the novelty claim. HUBERT provides a detailed layer-selection analysis which the authors did not mention, and 2405.08402 provides a checkpoint-wise analysis that covers a substantial part of this work.
Questions for authors
  1. Could the authors provide results on the iteration-2 probe score when iteration 2 clusters layers other than layer 10?
  2. Could the authors comment on how this protocol relates to HUBERT's own analysis and to 2405.08402?
  3. What is the GPU-hour cost of one full probe sweep, relative to one pre-training iteration?
Reviewer_th4MOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents wHuBERT, a HuBERT-inspired architecture developed as a general pre-trained encoder to serve various downstream tasks for 25 hz PPG and ACC data from consumer grade wearables. The motivation of the work is that training the second iteration of the HuBERT encoder from every candidate iteration 1 layer is expensive and resource-intensive. Thus, the authors propose a method that instead probes the iteration 1 encoder by fitting a classifier on each frozen layer using expert-scored labels, and choose the one with the best macro F1.

The authors evaluate their claims on a four-stage sleep staging task. They probe the selected layer on two public sleep-stage corpora, sleep-accel and BIDSleep, which have wrist ACC and expert stage labels, and evaluate model performance on an internal corpus. The authors report that the best-scoring layer is layer 10 or 11 of 12 across two pre-training lineages, and that label-free statistics such as RankMe do not recover these layers. In their experiments, a fine-tuned encoder and a distilled student score above a deployed baseline on the internal corpus, which the authors describe as a descriptive difference rather than an established improvement, since no paired interval can be computed.

Critical strengths

None

Critical weaknesses

The paper’s central claim is that probing the iteration 1 encoder selects a good clustering layer without training iteration 2 from each candidate layer. This seems to be untested as no iteration 2 model is trained from any layer other than the preconfigured layer 10, including the described Lineage B where the probe selected layer 11 but layer 10 was used. The probed encoders (lineages A and B) also differ from the encoder evaluated downstream (a separate lineage with different clustering, regularization, fusion layer, and size), which was never probed. To test the claim that the probe actually selects a good clustering layer, iteration 2 would need to be trained from n different layers of iteration 1, and each resulting encoder fine tuned for the downstream task and finally compared to the probe’s ranking of layers. The authors acknowledge the missing experiment in their limitations and state that the downstream comparison “does not isolate the layer choice" but this experiment must be included to validate the paper’s core claim of “low-cost layer selection”

The internal corpus, which is the sole source of labels for the sleep staging tasks as well as the basis for layer selection and evaluation, is almost entirely under described. Specifically, there is no description of the population the data is collected from, what proportion of the participants wore what devices, and how the corpus is generally assembled. Additionally, a participant is defined only as a study and number pair, and 60 of the 575 test recordings share study and participant numbers with ring files in the pre-training pool. Validation participants’ signals were also seen, unlabeled, in pre-training, and validation is where the probe’s best layer is chosen and its scores are reported. Without a clear description of the corpus I am to assess whether the splits are clean and whether the baseline comparison is fair.

Questions for authors

The experiments do not have convincing evidence that the proposed method holds.

--What is the composition of the internal corpus? Specifically: (a) how many recordings or participants come from each of the 9 devices, and which are rings versus watches (b) the participant population (c) how were the sleep stages scored and (d) what do the three collections supplying 92.9% of the pre-training pool contain?

--Was the deployed baseline trained on exactly the same labeled training split as the fine-tuned encoder?

--Were any iteration 2 encoders trained from a layer other than 10, whether or not reported? How was layer 10 originally chosen?

Reviewer_f2FYOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper adapts HuBERT to low-resolution PPG and accelerometer signals. Its main proposal is to use sleep-stage classification probes to choose the layer that supplies targets for the second pretraining stage. It also examines label-free selection criteria and reports sleep-staging results after fine-tuning and distillation.

Critical strengths
  • The detailed analysis across layers and training checkpoints helps characterize how sleep-stage information develops during pretraining.
  • The paper addresses a practical setting involving low-resolution wearable signals.
Critical weaknesses
  • The motivation and problem are not clearly established. The introduction does not adequately explain why HuBERT’s iterative training is suitable for these signals or what limitation of existing wearable methods it addresses. The paper also motivates general-purpose representations for multiple tasks, while selecting a layer using only sleep-stage labels. Relevant work such as HuBERT-ECG [1] should be discussed to clarify what is new about this adaptation and layer-selection study.

  • The main experiment does not test whether the proposed selection rule works. A layer that makes sleep stages easy to classify does not necessarily provide better targets for training the next model. The paper never trains second-stage models using different source layers and compares their performance.

  • The baselines do not establish the benefit of HuBERT pretraining. The downstream teacher has 186.1M parameters and uses raw sensor signals, whereas the supervised baseline has about 0.69M parameters and uses derived features. Their performance difference therefore cannot be attributed specifically to pretraining. There is no comparison with the same encoder trained from scratch or with another self-supervised method. Comparisons with representative contrastive and masked-reconstruction methods, using the same pretraining data and comparable model capacity and training budgets, would clarify the benefit of this approach. Lee et al. [2] is particularly relevant because it studies self-supervised learning for sleep staging using the same modalities at 25 Hz.

  • The test set overlaps with participants seen during unlabeled pretraining. Sixty of the 575 test recordings share study/participant identifiers with pretraining recordings, and 45 also have matching recording start times. This limits conclusions about generalization to unseen participants and nights. The authors should report results on a test set without these overlaps.

[1] Coppola et al. HuBERT-ECG: A self-supervised foundation model for broad and scalable cardiac applications. 2024. [2] Lee et al. Multimodal Self-Supervised Learning for Wearable Sleep Staging Using Photoplethysmography and Accelerometer Signals. ICASSP 2026.

Questions for authors
  • What specific problem in wearable self-supervised learning motivates HuBERT’s iterative target refinement?
  • Does the selected layer produce a better second-stage model than the final layer or another reasonable fixed layer?
  • How does wHuBERT compare with the same encoder trained from scratch and with representative contrastive or masked-reconstruction methods?
  • Do the downstream improvements remain after removing participant/night overlaps with pretraining?
  • How stable is the selected layer across independent training runs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.