- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
S-JEPA learns speech representations by predicting soft GMM posteriors using a KL-divergence objective. Training combines an initial MFCC-based GMM with online clustering of EMA encoder features, adaptive layer selection, and periodically switched EMA decay. The paper evaluates a 51.8M-parameter encoder on three SUPERB tasks and provides representation probes and predictive-entropy analyses.
Critical strengths
The paper targets a meaningful engineering problem: simplifying iterative clustering-based speech pretraining while retaining soft target assignments. The resulting encoder is evaluated across ASR, emotion recognition, and slot filling, demonstrating that the combined recipe learns useful representations.
Critical weaknesses
- W1: The comparisons do not isolate the effectiveness of the proposed objective. S-JEPA uses 83K hours of data, whereas several key baselines use 960 hours. Also, the compared encoders differ substantially in size. Although the paper acknowledges these differences, grouping models below 90M parameters does not constitute a matched comparison. Most smaller baselines are older methods, while DinoSR appears only in Appendix Table 4. Hence, the reported results do not establish an advantage over existing objectives under comparable model, data, and compute budgets.
- W2: The recipe introduces several changes without establishing which are necessary or beneficial. Section 6.1 acknowledges that online GMM updates, adaptive layer selection, and switched EMA decay are evaluated only together, while Appendix C simultaneously removes visible-position supervision and augmentation. Also, there is no controlled soft-target versus hard-target comparison, even though this distinction motivates the paper. Predictive entropy alone does not establish that soft supervision preserves useful acoustic ambiguity, since classifiers trained with hard targets also produce soft probabilities. These omissions leave both the mechanism and the incremental value of the method insufficiently supported.
Questions for authors
- Q1: Can the authors compare S-JEPA with established objectives such as HuBERT, data2vec, and DinoSR using the same encoder size, pretraining corpus, and comparable compute? A smaller shared training corpus would still provide a more informative comparison.
- Q2: Can the authors isolate soft versus hard targets while keeping the remaining recipe fixed, and provide ablations of adaptive layer selection and switched EMA decay?
- Q3: Can the authors report measured training time and clustering overhead against a relevant baseline? Removing offline reclustering simplifies orchestration, but the current evidence does not establish an overall computational advantage.