- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes finding groups of speakers with high ASR error without demographic labels. Each speaker is represented by normalized counts of phones from an automatic phone recognizer, and these vectors are clustered with a Bayesian Gaussian mixture. On TIMIT and a proprietary read-speech dataset, clusters from two phone recognizers are compared with race and race-gender groups for one commercial ASR system, using Welch's ANOVA, the largest pairwise effect size, and cluster-quality scores. The paper also reports that phone counts predict speaker-level WER better than demographic variables (cross-validated R^2 of 0.49 vs. 0.10).
Critical strengths
- Phone counts are easy to inspect and could help practitioners identify poorly served speaker groups.
- The WER prediction results suggest that these features contain useful information beyond demographic labels.
Critical weaknesses
- Label-free ASR speaker grouping already exists, but there is no direct comparison with the speaker-embedding approach. Comparing mainly with demographics does not establish what phone counts add (Dheram et al., 2022).
- The key results need stronger validation. For example, different sentence content can affect phone counts, while many small clusters create more chances to find an extreme error gap. These issues could change the conclusions about pronunciation and disparity (TIMIT).
- Overall, I do not find anything technically novel in this work.
Questions for authors
- Do the findings hold after controlling for sentence content and comparing methods with similar group sizes on held-out data?
- Can the authors provide predictions, scoring code, and the exact split for the reported zero phone error? Model card reports a different error metric on a different split (Model card).