- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
Authors analyze the effects of perturbation around a single content words and the effects of model certainty on word prediction. Find that perturbation around context around word affects certainty for Whisper class and wav2vec-ctc families of models.
This paper is highly detailed but is not advised for acceptance to the conference on grounds of: significance, data sparsity, and readability. I will propose my critique in by each category.
Critical strengths
Paper is fairly technically and in-depth in analysis of subject matter. Use of multiple model families shows consideration of potential biases in architecture design.
Critical weaknesses
Significance: The models under analysis are attention based models that are naturally trained to use speech context for inference. The author's discovery is a validation of a null assumption that context affects model certainty without any analytical hypothesis to reject. That is, it is difficult to understand what understanding of learning representations is being advanced by the thesis of this paper. There is thus limited scientific significance of this work to assist the community at large.
Further, the use of content words needlessly confines the potential analysis, regardless of null hypothesis being interrogated. The examples ("or,and,with...") are unstressed in English running speech; the phonetic reality naturally combines the words to general context of the onset and offset while also having a significant distribution in training corpus to be inferred through small segments. It is difficult to understand what nature of speech models is being investigated here that isn't simply a product of the internal language model acquired during neural training.
Finally, the wav2vec family of models has a strong bias towards LibriSpeech as is; there is a high likelihood that findings are more a consequence of the nature of the training and evaluation corpus than the actual behavior of speech models.
Data sparsity: By my count, evaluation samples are less than 100 utterances and alignment is used by a non-standard speech-to-text alignment model (MFA is still the main contender for accurate alignment selection, see recent work by Chodroff). Any variance in selection samples would have an exaggerated effect on findings. More data is needed, especially from other corpora, for findings to be reliable.
Readability: The paper attempts technical depth while using vague language and non-sequiturs that create difficulty in reading. Half of the abstract is unrelated to the main thesis the paper is trying to advance. There is limited impact form this work simply due to issues with understanding main thrust of the author's arguments in multiple points. Significant rewrite is suggested for future submissions to aid impact of paper.
Questions for authors
Why were given models selected for analysis?
What is the given hypothesis this paper is attempting to investigate against the null assumption of context dependency?
Why focus on only content words? Should not this application be used for several words and phrase structures to examine the degree the perturbation can be investigated?