- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes wHUBERT, a Conformer encoder adapted from HUBERT pretrained on wearable sensor signals for sleep staging. The main contribution is a protocol for selecting which iteration-1 layer of wHUBERT to use for clustering in iteration-2, by freezing iteration-1 checkpoints and fitting a probe on each of the candidate layers. This paper also tests whether label-free statistics can be used instead of this protocol, where the results indicate otherwise.
As the paper currently stands, it does not appear to be mature enough for publication, mostly due to the narrow scope of its contributions and lacking supporting evidence for its claims.
Critical strengths
- The authors are candid about the limitations of their work.
- The untrained-encoder control adds meaningful information as it shows that the interior peak in the probe comes from pretraining rather than architecture alone.
Critical weaknesses
- The authors did not test whether the layer choice matters. To validate this protocol, the authors should train iteration-2 encoders from layers other than 10, but that was not done. Hence there is no evidence that this protocol provides meaningful selection at all. This will be the single experiment that will go a long way to strengthen this paper. The authors also claim that the protocol is a computationally cheaper alternative, but provide no evidence at all to support it.
- The authors motivate this work by saying that "One clustering layer also forces a trade-off between tasks", suggesting that different tasks benefit from choosing different clustering layers. But the paper only tests one task. Uncited precendents also severely undercut the novelty claim. HUBERT provides a detailed layer-selection analysis which the authors did not mention, and 2405.08402 provides a checkpoint-wise analysis that covers a substantial part of this work.
Questions for authors
- Could the authors provide results on the iteration-2 probe score when iteration 2 clusters layers other than layer 10?
- Could the authors comment on how this protocol relates to HUBERT's own analysis and to 2405.08402?
- What is the GPU-hour cost of one full probe sweep, relative to one pre-training iteration?