- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The work focuses on ASR (automatic speech recognition) in a post-correction setting. The paper proposes a transcript selector for hypotheses generated by applying a fixed and frozen ASR model to the original and enhanced audio. Authors propose to train auxiliary neural models (experts) to estimate transcript error risk and select candidate transcripts, providing their scores and selections as additional information for the final selector. The selector receives features derived from the transcripts and audio, together with the expert information, and is trained to estimate each hypothesis’s error risk relative to a default ROVER. Overall, the selector decides which hypothesis to take or whether to keep the default ROVER transcript.
Critical strengths
The proposed method has improved the results.
The application of “Experts” has been shown to be valuable.
Critical weaknesses
Incremental improvement, limited methodological novelty: the core approach is supervised ranking of existing transcripts by predicted error risk. Advantages over established transcript selection and reranking methods are not clearly demonstrated. Only one ASR model is used for experimental evaluation, only several thousand test samples are used obtained from a limited number of recordings, no OOD/real-world evaluation, and important competitors are omitted (for example, a straightforward comparison with a LLM post-corrector). Ablation is limited. No computational overhead is discussed. The real-world applicability and transferability to new ASR are questionable.
The text is very hard to follow. The majority of paragraphs in the text sound overly shortened and unnatural. The methodology is hard to infer. The abstract and introduction should be revised to include general information and introductory context on the domain and task. The related work and limitations are discussed very briefly.
Questions for authors
-
Could you conduct a real-world or out-of-domain evaluation of your method? Namely, consider recording, e.g., from Zoom. At least you can consider other open-source speech datasets, e.g., Mozilla Common Voice (including other speech domains).
-
Do you have a confusion matrix on how many samples each method outperforms the others/tied-best/underperforms?
-
What are the complete inference latency/computational overhead, including candidate generation and acoustic scoring?
-
“Word error rate falls from 30.040% for reverse ROVER to 27.860%” - is the average metric calculated over all the recordings or subsets (balanced WER)? Could you please provide the metric using the other option? Is it possible to retain most of the method's positive gain while generating significantly fewer transcriptions?
-
Could you please compare your method with several competitors? For example, you can choose something from the following list (but you can use the other methods):
-
Jalalvand et al., 2015
-
Sato, Hiroshi, et al. "Learning to enhance or not: Neural network-based switching of enhanced and observed signals for overlapping speech recognition."
-
Ma, R., Gales, M.J.F., Knill, K.M., Qian, M. (2023) N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Space.
-
Chen, C., Hu, Y., Yang, C. H. H., Siniscalchi, S. M., Chen, P. Y., & Chng, E. S.. Hyporadise: An open baseline for generative speech recognition with large language models.
-
Iudin, A., Skripkin, M., Rogov, O. Y., & Korzh, D. Clarispeech: LLM-enhanced speech recognition post-correction.
-
Hori, T., Kocour, M., Haider, A., McDermott, E., & Zhuang, X.. Delayed fusion: Integrating large language models into first-pass decoding in end-to-end speech recognition.
-
Gong, X., Lv, A., Zhang, W., Wang, Z., Zhu, H., & Qian, Y. BR-ASR: Efficient and Scalable Bias Retrieval Framework for Contextual Biasing ASR in Speech LLM.
-
Hsu, Chan-Jan, et al. "Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactions.
-
Li, Haoyang, et al. "Training-Free Intelligibility-Guided Observation Addition for Noisy ASR."
- Appendix F1: What was the source of the data used for the additional evaluation?