acceptodds
Under review as a conference paper at ICLR 2027

Quality Before Quantity: What Governs the Gain from Self-Training in Low-Resource Speech Recognition

Abstract

Self-training with pseudo-labels is a commonly used strategy for domain adaptation, yet practitioners have no evidence with which to predict how much benefit it will provide, or how aggressively to filter, before running it. We study confidencefiltered self-training on speaker-disjoint and fully cross-corpus target splits, using an external test corpus that is never pseudo-labelled, oracle ceilings obtained from true target labels, and ten seeds per condition with paired non-parametric tests. We perform the full threshold sweep twice, using two different pools of unlabelled spontaneous Hindi 6.5 hours sampled speaker-disjointly from the target corpus, and 30 hours from a different corpus altogether and then perform a second self-training round on top of them. For both pools, self-training improves word error rate by 6.5 points absolute (15% relative), improves an external benchmark it never saw by 2.3 and 3.1 points, does not harm source-domain accuracy, and yields a student that outperforms its own teacher, trained on 128 hours of labelled speech, on every seed. Sweeping the threshold over the two pools reveals the same structure in both: once retained volume is controlled, downstream error is perfectly rank-ordered by the error rate of the retained pseudo-labels, and the optimum is interior, falling at the same threshold in both pools even though one is five times the size of the other and the teachers differ by fifteen points in quality. The relationship is strongly concave: almost all of the benefit comes from removing the worst labels, and filtering severely enough to retain only a small fraction of the data reverses the benefit even though those are the cleanest labels in the study. The second self-training round saturates: the improved teacher creates labels 2.8 points cleaner overall but only 0.4 points cleaner at the operating threshold, because iteration repairs precisely the labels that filtering had already discarded. We argue that filtering and iteration are substitutes rather than complements, that the pre-registered slope transfers in ordering but not in magnitude across corpora, and that quality drives the result, with diminishing returns, until volume becomes the limiting factor.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-training, pseudo-labeling, domain adaptation, speech recognition, low-resource ASR, confidence filtering, semi-supervised learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Wnj3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the mechanics of self-training with pseudo-labels for domain adaptation of low-resource ASR, specifically targeting the read-to-spontaneous speech shift in Hindi. It conducts an empirical study to determine how pseudo-label quality and quantity govern downstream WERs. It shows that once retained volume is controlled, downstream WER is perfectly rank-ordered by the error rate of the retained pseudo-labels, and the relationship is strongly concave. It also shows that, through a pre-registered prediction and a second iteration round, the structural ordering transfers across corpora but the linear slope does not, and that confidence filtering and iterative self-training act as substitutes rather than complements.

Critical strengths
  1. The empirical study to determine how pseudo-label quality and quantity govern downstream WERs is mathematically sound.
  2. The analysis of iteration saturation is insightful.
  3. Good experimental results. It demonstrates that a student trained on 10h labeled + 6.5h/30h pseudo-labeled data consistently outperforms its teacher trained on 128.5h of labeled data.
Critical weaknesses
  1. The scope is limited. As mentioned in the Limitations section, this study is conducted on Hindi only, adapting from a single read corpus using a single model size. While the two target pools provide some variation, it remains unclear whether the strong concavity and the specific threshold optimum generalize to other languages, different sizes of models, or different domain/acoustic conditions of the data.
  2. The exploration of filtering alternatives is limited. The paper mentioned soft-weighting but it only adopted hard thresholding. The comparison with hard thresholding against soft weighting or curriculum learning in the proposed framework was not conducted.
Questions for authors
  1. It is concluded that filtering and iteration are substitutes and advise against a second round. Since finding the optimal threshold requires evaluating multiple thresholds, how does the total compute cost of the threshold sweep compare to simply running two iterations of unfiltered self-training?
Reviewer_RZccOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studied the self-training ASR framework, whether a whisper-small teacher adapted on 128.5 h of labelled RESPIN read speech pseudo-labels spontaneous Hindi. Pool A contains 6.55 h of Lahaja speech whose speakers are disjoint from the Lahaja test half. Pool B contains 30 h of IndicVoices speech and is evaluated on the full, cross-corpus Lahaja set. A fresh LoRA student is trained from a 10 h labelled RESPIN subset plus pseudo-labels retained at one of five confidence thresholds. Each condition is repeated over ten seeds; FLEURS-Hindi and RESPIN test measure external and source-domain behavior. Oracle runs use true adaptation transcripts.

Critical strengths
  1. Replication across meaningfully different pools. The pools differ by 4.6 times in duration, by corpus, and by roughly 15 pseudo-label-WER points, yet the same broad threshold curve appears.

  2. Strong uncertainty reporting. Ten seeds, paired Wilcoxon tests, Holm correction, Cliff's δ\delta, seed-win counts, and mean/standard deviation are substantially better than typical ASR practice.

  3. Oracle and source-domain controls. The oracle quantifies remaining headroom, while RESPIN evaluation detects catastrophic forgetting. The pseudo-only control demonstrates the need for labelled anchoring.

  4. Useful practical signal. Removing the worst pseudo-label tail helps nearly for free; aggressive filtering eventually hurts. Reporting retained hours and estimated label quality is better than reporting an opaque threshold alone.

Critical weaknesses
  1. Quality is not isolated from quantity or sample composition. Changing the threshold jointly changes pseudo-label error, retained hours, and which utterances survive. The "volume-matched" analysis uses only four or five approximately matched threshold points with different examples, so the perfect rank correlations demonstrate a replicated association rather than a causal effect of label quality.

  2. The key quality variable is unavailable in deployment. Retained pseudo-label WER EE is computed using withheld reference transcripts, but no label-free estimator of EE is evaluated. The recommendation to reason about (E,H)(E,H) therefore remains difficult to apply to a genuinely unlabelled target pool, especially under confidence miscalibration caused by domain shift.

  3. The filtering-versus-iteration claim is underpowered. The only second-round experiment is run at a threshold where retained pseudo-label WER improves by just 0.0037, below the paper's estimated detectable downstream effect. The null result supports saturation for this pool and operating point, but not the broader conclusion that filtering and iteration are generally substitutes.

  4. External validity and subgroup coverage are narrow. The study uses one language, one Whisper size, one labelled source corpus, LoRA, and one confidence statistic. Filtering may also remove accents or speakers already poorly served by the teacher, improving aggregate WER while reducing coverage for difficult groups.

  5. Related-work positioning is incomplete. Established work on uncertainty-aware filtering, mean-token-confidence selection, label-free Whisper filtering, and large-scale pseudo-label refinement is missing. The paper should distinguish its threshold-sweep measurement contribution from methods that design stronger selectors, weighting schemes, or iterative pipelines.

Questions for authors
  1. On a fixed utterance set and fixed duration, what happens when supervision is changed among gold labels, teacher labels, and controlled label corruptions at prescribed WERs?

  2. How accurately can EE be estimated without reference transcripts, and does the downstream ordering remain when estimated rather than oracle EE is used?

  3. Does a second round help at unfiltered or looser thresholds, where the new teacher changes retained-label quality more substantially?

  4. Are results available for another language, Whisper size, full fine-tuning setup, or confidence estimator calibrated on target-domain speech?

  5. Which accents, districts, speakers, durations, and acoustic conditions are disproportionately removed as filtering becomes stricter?

  6. Would the authors narrow the title and conclusion from a causal claim that quality "governs" the gain to the replicated association identified by the current design?

Reviewer_WcQyOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates the relationship between pseudo-label quality, retained data volume, and downstream improvements from self-training in low-resource automatic speech recognition. The authors study confidence-filtered self-training using Whisper-small adapted from scripted Hindi speech to spontaneous speech, evaluating two unlabeled data pools of 6.5 and 30 hours. Experiments across ten random seeds demonstrate approximately 6.5 percentage points of absolute WER improvement over supervised fine-tuning, with additional improvements on an external evaluation dataset. The authors report that pseudo-label quality strongly correlates with downstream performance when retained data volume is approximately controlled, with diminishing returns and an intermediate confidence threshold producing the best results. Finally, a second round of self-training provides negligible additional improvements, leading the authors to conclude that confidence filtering and iterative self-training may offer overlapping benefits.

Critical strengths

S1. The paper addresses a practically relevant problem of understanding the trade-off between pseudo-label quality and quantity in low-resource speech recognition.

S2. The experimental setup includes speaker-disjoint and cross-corpus evaluations, oracle comparisons, and ten random seeds with statistical significance testing.

S3. The paper is generally well written and presents useful empirical observations regarding confidence filtering, diminishing returns, and iterative self-training.

Critical weaknesses

Based on my limited expertise: W1. Limited Methodological Novelty and Missing Comparisons: The paper primarily provides an empirical analysis of standard confidence-filtered self-training rather than introducing a new algorithmic or methodological contribution. The observation that better pseudo-labels improve downstream performance until insufficient training data remains is relatively intuitive and consistent with existing semi-supervised learning literature.

W2. Limited Generalizability and Weak Support for the Iterative Self-Training Conclusions: The study is restricted to Hindi speech recognition using a single Whisper-small architecture and one source corpus, making it difficult to determine whether the observed relationships generalize across languages, model scales, and domain shifts. More extensive experiments with independently selected teachers and multiple self-training rounds are necessary to substantiate this conclusion.

Questions for authors

See Weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.