acceptodds
Withdrawn from ICLR 2027

Recognition-Aware Selection of Enhancement-Derived Transcripts

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Different speech enhancers supply a frozen recognizer with complementary corrections and conflicting errors. CARE-ASR selects among the resulting transcripts by combining expert risk estimates with transcript agreement and acoustic support. It learns word-error differences and calibrates replacement against a complete default policy. We evaluate four English corpora, then apply the original meeting-speech policy without recalibration to 4,096 project-held-out AMI recordings from 16 meeting–speaker groups. Word error rate falls from 30.040% for reverse ROVER to 27.860%, a 2.180-percentage-point reduction (95% group-bootstrap interval [1.455, 2.919]; adjusted ). Fifteen groups improve, with more recovered words, fewer erasures and fewer insertions overall. Same-cohort controls examine the representation: expert inputs help under both pairwise and MSE training, whereas a matched nonlinear selector using only eight expert risks makes 210 more errors. When expert predictions for the AMI fitting sources are generated out of fold, the gain over reverse ROVER persists, and masking the explicit expert inputs adds 240 errors. The findings support learning from expert judgements together with candidate evidence to exploit enhancement diversity without updating the recognizer.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
speech recognition, speech enhancement, hypothesis selection, learning to rank, word error rate

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qRfLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The work focuses on ASR (automatic speech recognition) in a post-correction setting. The paper proposes a transcript selector for hypotheses generated by applying a fixed and frozen ASR model to the original and enhanced audio. Authors propose to train auxiliary neural models (experts) to estimate transcript error risk and select candidate transcripts, providing their scores and selections as additional information for the final selector. The selector receives features derived from the transcripts and audio, together with the expert information, and is trained to estimate each hypothesis’s error risk relative to a default ROVER. Overall, the selector decides which hypothesis to take or whether to keep the default ROVER transcript.

Critical strengths

The proposed method has improved the results.

The application of “Experts” has been shown to be valuable.

Critical weaknesses

Incremental improvement, limited methodological novelty: the core approach is supervised ranking of existing transcripts by predicted error risk. Advantages over established transcript selection and reranking methods are not clearly demonstrated. Only one ASR model is used for experimental evaluation, only several thousand test samples are used obtained from a limited number of recordings, no OOD/real-world evaluation, and important competitors are omitted (for example, a straightforward comparison with a LLM post-corrector). Ablation is limited. No computational overhead is discussed. The real-world applicability and transferability to new ASR are questionable.

The text is very hard to follow. The majority of paragraphs in the text sound overly shortened and unnatural. The methodology is hard to infer. The abstract and introduction should be revised to include general information and introductory context on the domain and task. The related work and limitations are discussed very briefly.

Questions for authors
  1. Could you conduct a real-world or out-of-domain evaluation of your method? Namely, consider recording, e.g., from Zoom. At least you can consider other open-source speech datasets, e.g., Mozilla Common Voice (including other speech domains).

  2. Do you have a confusion matrix on how many samples each method outperforms the others/tied-best/underperforms?

  3. What are the complete inference latency/computational overhead, including candidate generation and acoustic scoring?

  4. “Word error rate falls from 30.040% for reverse ROVER to 27.860%” - is the average metric calculated over all the recordings or subsets (balanced WER)? Could you please provide the metric using the other option? Is it possible to retain most of the method's positive gain while generating significantly fewer transcriptions?

  5. Could you please compare your method with several competitors? For example, you can choose something from the following list (but you can use the other methods):

  • Jalalvand et al., 2015

  • Sato, Hiroshi, et al. "Learning to enhance or not: Neural network-based switching of enhanced and observed signals for overlapping speech recognition."

  • Ma, R., Gales, M.J.F., Knill, K.M., Qian, M. (2023) N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Space.

  • Chen, C., Hu, Y., Yang, C. H. H., Siniscalchi, S. M., Chen, P. Y., & Chng, E. S.. Hyporadise: An open baseline for generative speech recognition with large language models.

  • Iudin, A., Skripkin, M., Rogov, O. Y., & Korzh, D. Clarispeech: LLM-enhanced speech recognition post-correction.

  • Hori, T., Kocour, M., Haider, A., McDermott, E., & Zhuang, X.. Delayed fusion: Integrating large language models into first-pass decoding in end-to-end speech recognition.

  • Gong, X., Lv, A., Zhang, W., Wang, Z., Zhu, H., & Qian, Y. BR-ASR: Efficient and Scalable Bias Retrieval Framework for Contextual Biasing ASR in Speech LLM.

  • Hsu, Chan-Jan, et al. "Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactions.

  • Li, Haoyang, et al. "Training-Free Intelligibility-Guided Observation Addition for Noisy ASR."

  1. Appendix F1: What was the source of the data used for the additional evaluation?
Reviewer_Y3UWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a method for combining multiple transcription variants into an improved transcript. It forms a fixed representation for each candidate transcript and trains a small network to predict its word-level difference from a reference transcript. The predicted scores are then used in a three-step rule for selecting the final candidate. The paper presents a set of candidate transcripts that includes direct-view transcripts, ROVER outputs, and expert-selected outputs. However, the construction of these candidates and the calibration procedure used to choose the final transcript are not clearly explained.

Critical strengths

The paper addresses a practical problem. Combining multiple transcription hypotheses could improve transcription quality, and the proposed method explores a learned scoring approach for selecting among them.

Critical weaknesses
  1. The paper’s contribution appears limited in novelty. As presented, the method scores candidate transcripts and selects among them. The paper does not make clear what is technically novel about this approach.
  2. The method and evaluation are insufficiently clear and motivated. The paper is difficult to follow and often reads like a list of components rather than a structured explanation. Key terms and design choices—including “voting transcript,” the ROVER outputs, the expert-selected outputs, the loss terms, and the calibration stage—are not adequately defined. The evaluation also lacks comparisons with relevant alternatives, such as LLM-based transcript scoring and generative error correction, making it difficult to assess the method’s value.
Questions for authors
  1. How are the direct-view transcripts, the two ROVER transcripts, and the expert-selected outputs constructed? What does “voting transcript” mean in this paper?
  2. Can you explain intuitively the loss terms in Eq. (3)-(4)?
  3. How does the calibration stage work, and how does it determine which candidate is selected?
  4. How does the method compare with LLM-based transcript scoring and generative error-correction methods and other baselines mentioned in the related work, such as HypR and N-best T5?
Reviewer_NfmAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

"Recognition-Aware Selection of Enhancement-Derived Transcripts" introduces a system named CARE-ASR which improves ASR performance through intelligently choosing between various audio-enhanced outputs. Their main contribution is the introduction of this selection procedure. They evaluate across CARE-ASR across 4 different datasets and compare against a variety of baselines including ROVER. Their largest performance improvement is on the AMI dataset.

Critical strengths
  1. Clear Motivation and Interesting Methodology - The authors explore the relevant problem of how to use multiple speech enhancement models to improve downstream ASR performance without re-training the existing model. The observation that different enhancement models can produce complementary corrections and errors is well motivated. The proposed approach of selecting among enhancement-derived transcripts using expert risk estimates, transcript agreement, and acoustic evidence is an interesting way to address this problem.

  2. Analysis and Experimental Validation - The authors conduct an extensive set of experiments across several datasets and compare their approach against various enhancement models, ROVER baselines, and learned selection methods. Specifically, the evaluation on a set of project-held-out AMI recordings results in evidence that the approach can outperform voting methods. The authors also perform ablations and control experiments to investigate the contributions of different features, training objectives, and expert predictions. The inclusion of statistical significance testing and confidence intervals for the additional AMI evaluation further strengthens the experimental findings.

Critical weaknesses
  1. Insufficient Clarity and Justification - The paper introduces a relatively complex framework without sufficiently explaining or motivating several of its key components. Concepts such as “experts,” “expert risks,” “reverse ROVER,” “set-context,” and “complete-default policies” are introduced without adequate intuition or explanation. Additionally, the authors do not sufficiently justify several design choices, including the use of 16 candidate slots, the selection of 110 features, and the construction of the 25-policy library. While some of these details are provided in the appendices, the main text would benefit from a clearer explanation of why these components are necessary and how they contribute to the overall approach. An illustrated example demonstrating how CARE-ASR selects a transcript would also improve the readability of the paper.

  2. Generalizability and Lack of Computational Complexity - One of the appealing properties of the proposed framework is its potential applicability to different ASR models without modifying their underlying weights. However, all experiments are conducted using Qwen3-ASR-0.6B, leaving it unclear whether the reported improvements generalize to other ASR architectures or stronger recognition models. Additionally, the complete system requires multiple enhancement models, repeated ASR inference, and additional acoustic scoring. However, the authors provide limited analysis of the associated computational overhead. This is particularly important given that several simpler selection methods achieve performance relatively close to CARE-ASR. As a result, it remains unclear whether the incremental improvements of the full framework justify its additional complexity and inference cost.

Questions for authors
  1. Given that the proposed framework does not require updating the underlying ASR model, can the authors clarify whether there is evidence that the approach generalizes to other ASR architectures? In particular an evaluation on stronger models would be interesting. Curious how the power of CARE-ASR changes when the recognizer produces fewer errors or less diversity as a result of enhancement?

  2. Can the authors provide a more detailed analysis of the computational overhead introduced by CARE-ASR compared with the methods they compare to? What are the implications for practical deployment?

  3. Can the authors elaborate on the motivation for the 16-candidate interface, 25-policy library, and the 110-feature set? How were these configurations selected? Is there evidence that such a specific configuration is necessary?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.