acceptodds
Under review as a conference paper at ICLR 2027

Speech recognizers carry the surrounding speech rate but use it less as they grow

Abstract

A probe can decode a variable from a network's activations without the network's behavior depending on it. Separating a variable the network carries from one it uses calls for a manipulation few variables permit: the variable moves while the input the decision is about stays fixed. Human speech perception supplies one. When the speech around a reduced function word such as “or” slows, listeners' reports of it fall from 79 to 33 percent though its audio is unchanged. A recognizer that did the same would drop a correctly transcribed word when only the speech around it slowed, a change that word error counts but cannot attribute to the context. We ran this manipulation on sixteen open checkpoints, twelve Whisper models across five sizes and four CTC recognizers. Whisper tiny through medium lower their belief in the frozen word, the log-probability lead of the transcript with the word over the one without it, as the surrounding speech slows. In the three smaller models the fall is roughly a third to half of the listeners' own, and the word drops out of the transcript. Three further function words and the CTC recognizers show the shift as well. Within Whisper the shift declines with size, persisting at 1.5B in two of three checkpoints, while the context's rate stays decodable from the same frozen frames at every size. Erasing that representation does not selectively remove the shift in Whisper and enlarges it in two of the three primary CTC recognizers. Together, this work separates a speech recognizer's representation from its use, which no benchmark that alters only the target, and no probe, can see. Our approach gives interpretability a stimulus-side test of use, complementary to probes and erasure, that costs one item set and no retraining.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, probing, representation versus use, behavioral testing, concept erasure, model scaling, inverse scaling, speech recognition, Whisper, CTC, speaking rate, rate normalization, speech perception, psychophysics, human-model comparison

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7b25OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

Authors analyze the effects of perturbation around a single content words and the effects of model certainty on word prediction. Find that perturbation around context around word affects certainty for Whisper class and wav2vec-ctc families of models.

This paper is highly detailed but is not advised for acceptance to the conference on grounds of: significance, data sparsity, and readability. I will propose my critique in by each category.

Critical strengths

Paper is fairly technically and in-depth in analysis of subject matter. Use of multiple model families shows consideration of potential biases in architecture design.

Critical weaknesses

Significance: The models under analysis are attention based models that are naturally trained to use speech context for inference. The author's discovery is a validation of a null assumption that context affects model certainty without any analytical hypothesis to reject. That is, it is difficult to understand what understanding of learning representations is being advanced by the thesis of this paper. There is thus limited scientific significance of this work to assist the community at large.

Further, the use of content words needlessly confines the potential analysis, regardless of null hypothesis being interrogated. The examples ("or,and,with...") are unstressed in English running speech; the phonetic reality naturally combines the words to general context of the onset and offset while also having a significant distribution in training corpus to be inferred through small segments. It is difficult to understand what nature of speech models is being investigated here that isn't simply a product of the internal language model acquired during neural training.

Finally, the wav2vec family of models has a strong bias towards LibriSpeech as is; there is a high likelihood that findings are more a consequence of the nature of the training and evaluation corpus than the actual behavior of speech models.

Data sparsity: By my count, evaluation samples are less than 100 utterances and alignment is used by a non-standard speech-to-text alignment model (MFA is still the main contender for accurate alignment selection, see recent work by Chodroff). Any variance in selection samples would have an exaggerated effect on findings. More data is needed, especially from other corpora, for findings to be reliable.

Readability: The paper attempts technical depth while using vague language and non-sequiturs that create difficulty in reading. Half of the abstract is unrelated to the main thesis the paper is trying to advance. There is limited impact form this work simply due to issues with understanding main thrust of the author's arguments in multiple points. Significant rewrite is suggested for future submissions to aid impact of paper.

Questions for authors

Why were given models selected for analysis?

What is the given hypothesis this paper is attempting to investigate against the null assumption of context dependency?

Why focus on only content words? Should not this application be used for several words and phrase structures to examine the degree the perturbation can be investigated?

Reviewer_a7yaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper examines how rate of speech of surrounding context influences a speech recognizer's behavior, motivated by prior reports that human listeners are less likely to recognize the word "or" when the surrounding context slows. The main experiment setup is the following:

  1. From LibriSpeech test-clean, the authors selected 44 segments containing the word "or". The authors call the word before "or", "or" itself, and the word after "or" the proximal window. The selected segments contain at least 2s of surrounding context before and after the proximal window.
  2. The authors manipulated the rate of speech of the surrounding context, and examined the log-probability / decoder output of the maniuplated audio vs the original audio.

The authors report on Whisper models, the slower the context, the less likely the model will predict "or" (Figure 2(a)). A similar trend is also observable on wav2vec2 CTC models but with different words (line 316: "and", "but", "that", "are").

The authors then report larger models are less sensitive to the surrounding context's slowing (Figure 3), in other words, they recognize the function word more often than tiny models.

The authors then showed the larger models still preserve the information of speech rate in their hidden representation, thus they are merely not utilizing them as much when predicting output tokens.

Finally, the authors attempted to erase speech rate features from the models, and examined how the outputs change. However, it appears this is largely inconclusive.

Critical strengths

The authors designed careful control experiments, and reported a large amount of experimental results.

Critical weaknesses
  • Significance: It is not clear to me what this paper wants to achieve other than reporting some results on a somewhat expected behavior. ASR models are trained on natural speech, which tends to have consistent speech rate. It's not surprising that they make use of the speech rate in recognition. Not to mention that English phonology leans heavily on duration for distinguishing vowels, placing stress, etc.
  • Soundness: This paper reports that larger Whisper models are less sensitve to the surrounding context's slowing. However, it's worth noting earlier smaller Whisper models are trained differently than later larger ones. In addition to additional training data, OpenAI reports that large-v2 used SpecAugment. Although not entirely clear from publicly available information, there is a possibility that time warping (part of SpecAugment) was used in large-v2 training, which can be another reason why larger Whisper models were less sensitive.
Questions for authors

None.

Reviewer_uSuyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether the speed of surrounding speech affects word recognition in speech models. The authors reduce the duration of a target word, such as “or,” and then change the speed of the surrounding speech. The target word and its neighboring words stay unchanged across the different speed conditions. They test Whisper and CTC models. The main Whisper models become less likely to recognize the target word when the surrounding speech slows down. This effect is generally weaker in larger Whisper models, but probes can still identify the speed conditions from their hidden states. The authors argue that having information in a model does not mean the model uses it when making decisions.

Critical strengths
  1. The paper asks a meaningful question. A model may have information about speech rate, but this does not mean it uses that information to recognize words. The paper studies this difference. This is useful because it helps us understand how models make decisions, rather than only measuring how many words they recognize correctly.
  2. The experiment is clearly designed. The authors keep the target word and its neighboring words unchanged while changing the speed of the surrounding speech. This makes it easier to study how the surrounding speech affects recognition. They also check whether the results come from changes in recording length or audio processing.
  3. The authors examine both model scores and generated transcripts. When the surrounding speech slows down, some Whisper models leave out the target word. This shows that the effect is not just a change in scores. Tests with other words also suggest that the finding is not limited to “or.”
Critical weaknesses

1.The constructed speech may be different from what the models normally hear. Shortening the target word to a quarter of its original length while slowing down the surrounding speech may create an unnatural recording. The existing controls do not fully address this issue. Tests with naturally reduced words and smaller speed changes would help show whether the findings extend beyond these artificial samples. 2.The reason for the weaker effect in larger models is still unclear. Larger models may be more robust to the edited audio, so this pattern may not reflect how they use speech rate in natural speech. Some checkpoints also differ in their training setup, which makes the role of model size harder to judge. 3.The CTC results are less clear and need further checking. Some control scores change substantially, making the target effect harder to interpret. The authors apply an additional correction, but it is unclear whether the reported intervals include uncertainty from fitting it. They also report a decoding error and try to address it through re-scoring. Results from a corrected decoder would provide stronger support for the transcript findings.

Questions for authors

1.The edited samples may be out of domain for the models. Could larger models show a smaller effect because they handle these samples better? Does the effect also appear with naturally reduced words, without artificially shortening the target? 2.For CTC, do the intervals for the corrected median include uncertainty from fitting the correction? Appendix B reports a repeated-letter decoding error. Could the authors fix the decoder and check whether the WER results and the clean-versus-erased comparisons remain similar?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.