acceptodds
Withdrawn from ICLR 2027

ASR Evaluation with Phonetic Embeddings

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Evaluating a speech system using mean or median Word Error Rate (WER) can obscure meaningful differences in performance among subgroups within the evaluation dataset, and among groups of speakers who will interact with the system. Prior work has shown that these disparities can be significant, and pose a challenge to adoption. However, to measure fine-grained disparities, developers typically rely on datasets that are annotated with speaker demographics. Such annotations are costly, noisy, and not causally connected to speech behaviors that trigger recognition error. In this work, we propose a new data-driven technique for measuring error disparity. It is fully automatic, derived from meaningful speech behavior, and directly correlated with drivers of error. Phonetic embeddings are a data-driven representation of speech that can be used to cluster speakers and measure error disparity among clusters. Our technique addresses the gaps in current evaluation methods with a speaker representation that directly correlates with error, and a fully automatic method for disparity analysis. In an evaluation of an off-the-shelf commercial ASR system, we show that this method finds larger regression coefficients, performance gaps with similar or greater effect size, and better cluster quality, compared to demographic methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Evaluation, Speech Recognition, Acoustic Representation, Responsible AI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RU2cOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes finding groups of speakers with high ASR error without demographic labels. Each speaker is represented by normalized counts of phones from an automatic phone recognizer, and these vectors are clustered with a Bayesian Gaussian mixture. On TIMIT and a proprietary read-speech dataset, clusters from two phone recognizers are compared with race and race-gender groups for one commercial ASR system, using Welch's ANOVA, the largest pairwise effect size, and cluster-quality scores. The paper also reports that phone counts predict speaker-level WER better than demographic variables (cross-validated R^2 of 0.49 vs. 0.10).

Critical strengths
  1. Phone counts are easy to inspect and could help practitioners identify poorly served speaker groups.
  2. The WER prediction results suggest that these features contain useful information beyond demographic labels.
Critical weaknesses
  1. Label-free ASR speaker grouping already exists, but there is no direct comparison with the speaker-embedding approach. Comparing mainly with demographics does not establish what phone counts add (Dheram et al., 2022).
  2. The key results need stronger validation. For example, different sentence content can affect phone counts, while many small clusters create more chances to find an extreme error gap. These issues could change the conclusions about pronunciation and disparity (TIMIT).
  3. Overall, I do not find anything technically novel in this work.
Questions for authors
  1. Do the findings hold after controlling for sentence content and comparing methods with similar group sizes on held-out data?
  2. Can the authors provide predictions, scoring code, and the exact split for the reported zero phone error? Model card reports a different error metric on a different split (Model card).
Reviewer_nUDyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Measuring differential performance of ASR systems across speaker groups is a necessary supplement to median/mean WER, but these measurements require costly human annotation in the cases where that metadata is unavailable. The authors here propose inducing pronunciation vectors in an unsupervised manner using normalized proportions of transcribed phones. They find indeed that these pronunciation vectors modulate ASR performance.

Critical strengths
  • The features are easy to interpret, and don’t require human annotation.

  • The authors identify that disparate ASR performance is an issue

Critical weaknesses
  • This approach feels quite circular, using automatic (phone) transcriptions to predict the error in automatic (word) transcriptions.

  • Phonetic realization is but one axis of variation, and one that imperfectly maps on to accent, race, etc, we even see that people from the same ’accent’ (which an approach like this would be better suited to differentiate) can carry with them wildly different features. Where then, are any baselines/controls of these labeled features? Seems like a missed opportunity to explore how articulatory features modulate ASR performance.

  • The TIMIT numbers don't line up. Figure 3 shows a cohort-versus-rest WER gap above 0.08, while §4.4 gives the highest cohort mean as 0.077. The gap cannot exceed that mean. The authors should reconcile the figure and text, though this discrepancy doesn't by itself invalidate Table 5.

  • Working with read speech corpora doesn't necessarily fix word choice- it seems like TIMIT speakers read different prompts.

Questions for authors

Can you reconcile Figure 3 and §4.4? Does the predictive advantage persist with matched prompts or a reference-text control?

Reviewer_u9GeOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses the issue of evaluating the quality of speech transcription for outlier groups of speakers. Annotation with speaker demographics doesn't capture the speech patterns that may cause degradation in recognition performance. Authors propose a data-driven method of determining the speech behaviors that directly lead to differences in errors via phonetic embeddings. Authors show how this data-driven approach, compared to using demographics, leads to discovering larger performance gaps and yields better clusters of speakers.

The experimental results demonstrate the speaker representation of normalized phone vectors is robust to the 1. phone model: high-quality, en-only phone recognizer and less accurate multilingual phone recognizer, 2. on two different datasets TIMIT and in-house English evaluation set. The results show that the phonetic speaker clusters explained the WER differences 5x more than the demographics. Qualitative analysis showed that some clusters corresponding to linguistic phenomena and thus provide some level of interpretability.

Critical strengths

The paper demonstrates a data-driven way to assess weaknesses in a speech transcription system beyond raw WER and slicing by speaker demographics through a phonetic embedding representation of speakers. The results are clear, multiple systems and datasets are tested and the results appear significant.

Critical weaknesses

One of the major datasets is an internal. Reproducibility is reduced due to the lack of a full script to reproduce results, although implementations of the key algorithms are available.

The topic area may be considered somewhat niche since this application is for the particular domain of speech processing and on improving the evaluation of models.

Questions for authors

How much data per speaker is needed for a reliable vector of phones?

With more data, phonetic context could be used; have you looked at how much more of the errors can be explained by moving to bigram or trigram phone representations?

How well does this approach generalize to spontaneous speech where enunciation can falter and other ASR models like Whisper or Nemotron?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.