wHuBERT: Layer Selection for Iterative Masked-Prediction Pre-training on Low-Resolution Wearable Biosignals

Applications for wearable biosignals increasingly use the foundation model paradigm, where models are pre-trained without labels. HuBERT, originally developed for speech, requires two pre-training phases, the first one with coarse clustered pseudo-labels, and the second one with clusters derived from the representations of an intermediate layer from the first phase. The selection of the correct layer for the second iteration is crucial for a well-performing model. However, accurate performance metrics can only be obtained after the two pre-training phases and one fine-tuning phase for a downstream task. This represents a large commitment of compute before any concrete metric is available. One clustering layer also forces a trade-off between tasks and undercuts the "train once, apply anywhere" premise. We measure the layer selection effect on wHuBERT, a Conformer encoder pre-trained on photoplethysmography and accelerometry sampled at 25 Hz, the rate consumer wearables use. We propose a cheap probe-based protocol, applied to every layer at regular checkpoints, that selects the best-scoring layer for the given task. Across 2 independent runs, the final-checkpoint best-scoring layer for sleep staging lies just below the top, rarely at the final layer. We also computed 7 label-free statistics, including estimates of effective rank, but show they are not reliable for layer selection. We verify these results on two public sleep-stages corpora. We also fine-tune and distill an iteration-2 encoder and both the teacher and student score above the baseline from a consumer device. The protocol thus offers low-cost layer selection for foundation models on low-resolution biosignals.