acceptodds
Under review as a conference paper at ICLR 2027

S-JEPA : Soft Clustering Anchors for Self-Supervised Speech Representation Learning

Abstract

Self-supervised speech encoders are predominantly trained by predicting discrete hard cluster IDs at masked positions, a recipe that collapses acoustic ambiguity at category boundaries and requires interrupting training to re-cluster the entire corpus between iterations. We introduce S-JEPA, a JEPA-style encoder-predictor pair trained to match the soft posteriors of a Gaussian Mixture Model at masked positions via KL divergence. Training runs as one continuous optimization trajectory in two phases: a fixed GMM over MFCC features, then an online GMM over encoder features, with the input layer selected adaptively from a label-free signal, so the re-cluster step runs online rather than offline and the choice of which transformer layer to cluster on is no longer hand-tuned. Under the SUPERB protocol, S-JEPA achieves the lowest WER among evaluated SSL methods below 90M parameters and matches HuBERT-Base on emotion recognition at roughly half its parameter count, with no offline re-clustering and no pretrained teacher. Pre-training data differs across the compared methods, so this is a comparison at matched parameter count rather than matched data or compute. An analysis of the predictor’s per-frame entropy on held-out speech reveals a bimodal distribution with a substantial minority of frames near the entropy of a perfect two-cluster tie, showing that the objective retains inter-cluster mass on frames where a hard target would have to discard it.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
JEPA, Representation Learning, Speech

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_dJ2rOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

S-JEPA learns speech representations by predicting soft GMM posteriors using a KL-divergence objective. Training combines an initial MFCC-based GMM with online clustering of EMA encoder features, adaptive layer selection, and periodically switched EMA decay. The paper evaluates a 51.8M-parameter encoder on three SUPERB tasks and provides representation probes and predictive-entropy analyses.

Critical strengths

The paper targets a meaningful engineering problem: simplifying iterative clustering-based speech pretraining while retaining soft target assignments. The resulting encoder is evaluated across ASR, emotion recognition, and slot filling, demonstrating that the combined recipe learns useful representations.

Critical weaknesses
  • W1: The comparisons do not isolate the effectiveness of the proposed objective. S-JEPA uses 83K hours of data, whereas several key baselines use 960 hours. Also, the compared encoders differ substantially in size. Although the paper acknowledges these differences, grouping models below 90M parameters does not constitute a matched comparison. Most smaller baselines are older methods, while DinoSR appears only in Appendix Table 4. Hence, the reported results do not establish an advantage over existing objectives under comparable model, data, and compute budgets.
  • W2: The recipe introduces several changes without establishing which are necessary or beneficial. Section 6.1 acknowledges that online GMM updates, adaptive layer selection, and switched EMA decay are evaluated only together, while Appendix C simultaneously removes visible-position supervision and augmentation. Also, there is no controlled soft-target versus hard-target comparison, even though this distinction motivates the paper. Predictive entropy alone does not establish that soft supervision preserves useful acoustic ambiguity, since classifiers trained with hard targets also produce soft probabilities. These omissions leave both the mechanism and the incremental value of the method insufficiently supported.
Questions for authors
  • Q1: Can the authors compare S-JEPA with established objectives such as HuBERT, data2vec, and DinoSR using the same encoder size, pretraining corpus, and comparable compute? A smaller shared training corpus would still provide a more informative comparison.
  • Q2: Can the authors isolate soft versus hard targets while keeping the remaining recipe fixed, and provide ablations of adaptive layer selection and switched EMA decay?
  • Q3: Can the authors report measured training time and clustering overhead against a relevant baseline? Removing offline reclustering simplifies orchestration, but the current evidence does not establish an overall computational advantage.
Reviewer_kWnSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

S-JEPA is a 51.8M-parameter (6-layer) self-supervised speech encoder trained in a two-phase pass by matching soft posteriors from a diagonal-covariance Gaussian Mixture Model (GMM) via KL divergence. Phase 1 uses a frozen 100-component GMM fit on 39-dim MFCCs, while Phase 2 switches to an online 500-component EMA-updated GMM over an EMA teacher encoder, adding effective-rank (erank\text{erank}) adaptive layer selection and periodically switched teacher EMA decay (αt∈{0.999,0.9999}\alpha_t \in \{0.999, 0.9999\}). Pre-trained on ~83,000 hours of English speech (LibriLight + Granary), S-JEPA achieves 12.10% WER on LibriSpeech test-clean and 64.83% accuracy on Emotion Recognition (SUPERB frozen-encoder protocol), and exhibits a bimodal per-frame predictor entropy distribution on held-out speech.

Critical strengths
  1. Practical Single-Pass Pipeline & Structured Uncertainty: Combining online GMM sufficient-statistic updates with label-free erank\text{erank} layer selection eliminates the full-corpus feature extraction and disk relabeling pass required between HuBERT/WavLM iterations. Additionally, the entropy analysis (Section 5.2) nicely shows that the predictor captures structured two-way cluster competition near 1 bit (36.6% of frames >1>1 bit) rather than uniform collapse.
  2. Transparent Presentation: The paper is clearly written and forthright about its scope, explicitly noting that its comparisons are parameter-matched rather than data-matched (Section 4, Lines 299–304) and that its empirical findings are correlational (Section 6).
Critical weaknesses
  1. Incremental Novelty Without Controlled Ablations: Single-pass online discretization/clustering is already well-established in speech SSL via trainable Gumbel-Softmax/VQ codebooks (vq-wav2vec, wav2vec 2.0) and online EMA clustering over an EMA teacher (DinoSR, which reaches 4.71% WER in Table 4). As noted in Section 2 (Lines 128–155), S-JEPA's delta over DinoSR is primarily swapping hard online kk-means for soft diagonal GMM posteriors and replacing multi-layer clustering with single-layer erank\text{erank} selection (while still needing an MFCC warmup and kk-means+EM re-initialization at the Phase 1$\to$2 transition). Crucially, as admitted in Sections 6 and 6.1 (Lines 462–470), none of these components are ablated—there is no controlled comparison of soft GMM KL loss vs. hard online assignments, erank\text{erank} selection, switched EMA decay, or the mid-Phase-2 loss/augmentation interventions (Appendix C).
  2. Severely Confounded Pre-training Data Scale (83,000h vs. 960h): The claim of a new sub-90M Pareto frontier (Table 1, Figure 2) is confounded by an 86×86\times data disparity. S-JEPA is trained on ~83,000 hours, whereas every sub-90M baseline (DistilHuBERT, DeCoAR 2.0, wav2vec, vq-wav2vec) and the HuBERT-Base and WavLM-Base checkpoints in Table 1 and Table 2 (Section 4, Lines 299–304; Appendix H, Line 948) were trained on 960 hours of LibriSpeech (WavLM-Base+ at 94k hours is relegated to Appendix J, Table 4). Without a 960h S-JEPA run, downstream performance cannot be separated from data scale.
Questions for authors
  1. Controlled 960h Baseline & Soft vs. Hard Ablation: Can you provide a controlled run on LibriSpeech 960h (or a 100h–960h subset) comparing S-JEPA's soft GMM KL objective against an identical pipeline using hard online cluster assignments (cross-entropy on arg⁡max⁡kqt,k\arg\max_k q_{t,k}) and fixed EMA decay?
  2. Comparison Against Data-Matched WavLM-Base+: Why was WavLM-Base (960h) used instead of WavLM-Base+ (94k hours) in Table 1 and the Table 2 probing experiments (Appendix H, Line 948)? How does S-JEPA compare against WavLM-Base+ on the Table 2 probes?
  3. Layer-Switch Stability & Mid-Phase-2 Interventions: When erank\text{erank} switches the target layer from ℓ⋆=2\ell^\star = 2 to ℓ⋆=4\ell^\star = 4 mid-Phase 2 (Appendix G), how does the online GMM avoid distribution shift without re-initialization? Also, at what step were the visible-position loss and denoising manually disabled (Appendix C), and what happens if they remain active?
Reviewer_CPYAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The proposed framework replaces HuBERT's hard cluster-based training with soft GMM posteriors and combines KL-divergence-based mask prediction. Experimental results on SUPERB demonstrate competitive performance compared to existing speech SSL methods.

Critical strengths
  1. The paper addresses the limitations of hard clustering in self-supervised speech representation learning.

  2. The proposed framework integrates soft clustering with online updates and demonstrates promising performance compared to existing speech SSL methods.

Critical weaknesses
  1. S-JEPA is pretrained on about 83,000 hours of speech, whereas other methods are pretrained only 960 hours. Although the authors acknowledge this discrepancy, a fair comparison under matched pretraining data and computational budgets is necessary to establish the effectiveness of S-JEPA.

  2. The proposed training framework shares substantial similarities with data2vec and data2vec 2.0, particularly in its use of an EMA teacher, masked prediction, and teacher–student learning. Although S-JEPA replaces continuous representation regression with soft GMM posterior matching, the paper does not adequately establish whether this modification provides meaningful advantages over existing approaches.

  3. Neither data2vec nor data2vec 2.0 is included in the experimental comparisons, despite their close methodological relevance. Without these comparisons under comparable training conditions, it remains unclear whether the proposed framework offers a substantial improvement over established teacher–student SSL methods.

  4. Moreover, despite the availability of numerous established speech SSL baselines and a broader set of SUPERB tasks, the paper evaluates only a limited selection of methods and three downstream tasks. I encourage the authors to conduct more comprehensive evaluations across a wider range of SSL models and downstream tasks.

  5. The paper also does not evaluate full fine-tuning performance, which is particularly important for assessing the practical effectiveness of pretrained speech representations.

  6. Although S-JEPA combines soft GMM supervision, online clustering, adaptive layer selection, and switched EMA decay, their individual contributions are not evaluated. In particular, the paper does not compare soft and hard clustering under identical conditions, despite this being its central motivation. Similarly, adaptive layer selection and switched EMA decay are not compared against fixed alternatives. Without these ablations, it remains unclear which components are responsible for the reported improvements.

  7. The evaluation is limited to a single 51.8M-parameter model, without experiments at Base or Large scales. Although the paper emphasizes parameter efficiency, it remains unclear whether the proposed method can scale effectively to larger architectures or maintain its advantages over established SSL methods at comparable model sizes. I encourage the authors to evaluate S-JEPA at Base and Large scales to demonstrate its scalability and generality.

Questions for authors

Please see the concerns detailed in the Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.