acceptodds
Under review as a conference paper at ICLR 2027

Audio LLMs Know When They Can't Hear You

Abstract

Audio large language models (LLMs) allow users to interact with the model through speech. When an input recording is too degraded, the model may misinterpret the user's query and respond based on an incorrect transcription. In this paper, we study model-conditional transcription reliability: whether an Audio LLM can recognize when its own audio transcription is unreliable. We first prompt the Audio LLM to assess whether its own transcription would be reliable, and find that the model is a poor judge of its own transcription reliability: in most cases, it predicts that its transcription will be reliable. Next, we find that existing approaches, including speech quality predictors, audio LLM generation uncertainty, and transcript-conditioned WER estimation, provide only limited signals for detecting transcription failures. In contrast, we discover that transcription reliability is strongly represented in the model's audio-encoder representations. Based on this observation, we devise a lightweight reliability predictor that operates on representations extracted by the frozen audio encoder and predicts the reliability class before generation. The reliability predictor can trigger a clarification request from the user when their voice query is predicted to be unreliable, while allowing reliable queries to proceed without modifying the underlying Audio LLM. Our predictor achieves 81.10% in-domain and 78.09% cross-domain macro-F1 scores, outperforming the strongest baselines by 10.33 and 11.93 points, respectively. Finally, we show that reliability labels can transfer across Audio LLM families, and that transfer performance is closely related to the alignment of their model-specific reliability boundaries.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio Large Language Models, Transcription Reliability, Robust Speech Recognition, Reliability Prediction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9X8yOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether an Audio LLM's transcription reliability can be predicted before language-model decoding. It constructs model-specific label banks by searching for SNR values at which transcription WER crosses 0%, 10%, and 30%, then trains learned temporal pooling and a small classifier over a frozen audio encoder to predict four reliability classes. On Qwen2-Audio, the reported macro-F1 is 81.10% in-domain and 78.09% cross-domain, with additional results on Phi-4-Multimodal and MOSS-Audio. The paper also examines prompted self-assessment, encoder-depth and pooling ablations, and reuse of another model's label bank. Boundary shifts estimated from 5% of shared speech-corruption pairs substantially improve the two reported transfers to MOSS. The intended application is to identify potentially unreliable voice inputs before responding.

Critical strengths
  1. A useful problem with a simple, potentially practical solution. Predicting transcription reliability before decoding could support early clarification decisions without changing the underlying Audio LLM. The reported improvements are consistent across three model families and both evaluation settings (Tables 1, 3, and 4). For Qwen2-Audio, the macro-F1 improvements over the strongest baseline are 10.33 and 11.93 points. Appendix C also describes disjoint speech speakers/utterances, noise recordings, and room environments across the in-domain splits. These are meaningful positive results for a lightweight prediction head, although the deployment implications still depend on the validity and scope of the labels.

  2. The label-bank reuse study adds value beyond a standard encoder probe. The authors investigate how model-dependent corruption boundaries affect transfer and propose a simple correction using limited target-model measurements. Table 7 reports that per-boundary median alignment reduces the accuracy deficit for MOSS <- Qwen2 from 9.69 to 1.70 points and for MOSS <- Phi-4 from 12.86 to 2.05 points. If confirmed under a clearly separated calibration/evaluation protocol, this would make the approach substantially less expensive to adapt to another Audio LLM.

Critical weaknesses
  1. The validity of the per-example evaluation labels is not sufficiently established. Section 3.1 and Appendix C estimate three critical SNRs with binary search under an approximate monotonicity assumption, then sample training and fixed evaluation waveforms from the resulting intervals. The description does not explicitly establish whether every fixed validation/test waveform is transcribed again and labeled using its realized WER. Ordered boundaries and margins around them do not by themselves establish that WER stays within the assigned class throughout each interval. If test labels are inherited from those intervals, the reported metrics could partly measure agreement with the label-construction approximation rather than actual transcription reliability. This matters directly to the central claim. In addition, the evaluation follows filtering that removes clean/reverberation failures and ambiguous boundary cases, so the conclusions should be scoped to this selected population until performance on excluded or boundary-adjacent cases is characterized. I would like a direct audit against realized WER before treating the reported gains as convincing evidence of failure detection.

  2. The cross-model transfer evidence needs a clearer and internally consistent protocol. Section 4.2 does not specify clearly which split supplies the 5% target-labeled pairs, whether these pairs are disjoint from the reported evaluation set, and whether all transfer/alignment variants use the same fixed target test population after filtering. This is an ambiguity, not evidence that leakage occurred. There is also an apparent discrepancy within Figure 5: the middle panel places the Qwen2/Phi-4 mismatch at approximately 2.7 dB and the Phi-4/MOSS mismatch at approximately 3.45 dB, whereas the right panel's full-pair estimates are approximately 2.35 and 3.95 dB, respectively. If these use different pair sets or definitions, that needs to be stated; otherwise the figures should agree under Eq. (6). Because boundary mismatch and low-cost alignment form a central contribution, these details materially affect my assessment.

Overall, the method and reported gains are promising, but these two unresolved validation issues lead me to weak rejection at present. A convincing realized-WER audit and clarification of the transfer protocol could change my recommendation.

Post-rebuttal update (19 Nov 2026):

The revised evaluation against realized WER, the clarified transfer protocol and the coverage/WER analyses address my main concerns. The latest revision also includes the calibration subset sizes and the five-run variability results requested in the discussion. I am keeping my rating at 3 (weak acceptance). The evidence supports transcription reliability; broader audio-understanding failure detection remains outside the demonstrated scope. This is my final recommendation after the rebuttal, and I have no further questions.

Questions for authors
  1. Clarification and validation of the existing results: Are the labels used to score Tables 1, 3, and 4 computed from a fresh transcription of each fixed evaluation waveform, or inherited from its sampled SNR interval? Please report the agreement between interval labels and realized-WER classes on an independently checked subset (including samples near the retained interval edges), and whether the main comparison changes when scored against realized WER.

  2. Clarification of the existing transfer experiment: Which split and how many shared pairs supply the 5% alignment data? Are they disjoint from evaluation, and is one unchanged target-model test set used for direct, aligned, and target-label-bank training? Please also reconcile the middle/right-panel mismatch values in Figure 5 and report variability of the aligned-transfer results over independently sampled 5% subsets.

  3. Scope of the proposed clarification use case: On the existing fixed tests, what fraction of inputs would proceed without clarification, and what is the realized transcription-error rate among those accepted inputs at a specified operating point? Figure 6 provides useful class-conditional error rates, but a coverage/error trade-off would clarify whether the predictor supports the proposed decision. This is a request to analyze the existing predictions, not a request for a new user study.

5 comments
Reviewer_9X8yOpenReview ↗

Note on AI tool use during my review

Following the PCs' request, I'm adding a note about how AI tools were used during my reviewing process. I discussed this paper with ChatGPT (GPT-6-Astra) while working on my review. I used it as a sounding board: I shared my own reading notes — mainly the label-validity concern about the SNR-interval label construction in Sec 3.1 / Appendix C, and the protocol ambiguities in the cross-model transfer experiment (Sec 4.2, Figure 5) — and asked it to check whether my criticisms were technically sound and whether I had overlooked obvious counterarguments. It also helped me polish the wording of the review text. My assessment of the paper, the rating, and all specific claims in the review are my own: I verified every cited detail (the label construction procedure, Table 7, the Figure 5 mismatch values, etc.) directly against the manuscript. The AI discussion helped me articulate the label-validity concern more precisely, but it did not change my overall assessment. My questions for the authors in the review stand as written, and I look forward to their responses.

AuthorsOpenReview ↗

Response to Reviewer 9X8y

We are grateful to the reviewer for the detailed and constructive review, for recognizing the practical value of our approach, its consistent gains across three model families, and the contribution of the label-bank transfer study, and for identifying precisely which validations would make the evidence convincing. Below we respond point-by-point and summarize the corresponding updates in the revised manuscript.

1. Validity of the evaluation labels and scope of the evaluation (W1, Q1)

(a) Audit against realized WER. We thank the reviewer for this important point. Every fixed test waveform was transcribed by the target Audio LLM, and its WER was stored. However, the labels used to score the tables of the original submission were inherited from the SNR interval from which each test waveform was generated. We audited the agreement between this interval label and the reliability class of the realized WER of each test waveform:

Audio LLMIn-domain testCross-domain test
Qwen2-Audio-7B-Instruct94.8%93.6%
Phi-4-Multimodal-Instruct95.9%95.0%
MOSS-Audio-8B96.9%97.1%

The two labels agree for 93.6% to 97.1% of test waveforms, and 92.9% of the disagreements are between neighboring reliability classes.

Following the reviewer's suggestion, the label of every test example in the revised manuscript is assigned from the realized WER of the target Audio LLM on that example (Section 3.1), and all results are rescored with the same predictions and the same fitted baseline thresholds: the main comparison (Tables 1, 3, and 4), the confusion matrices (Figure 6), the label-transfer results (Figure 5), and the ablations (Appendix L). The analyses added in the revision, including the encoder comparison (Table 6), the evaluation across the class-specific SNR intervals (Figure 7), the coverage analysis (Figure 8), and the decoder-layer comparison (Figure 11), are likewise evaluated with labels assigned from the realized WER. The changes in the results are minimal, and our predictor remains the best method on every metric for all three Audio LLMs in both evaluation settings.

To characterize the labels and the predictor near the interval edges, we added an evaluation in Appendix F (Figure 7). Each in-domain test example is generated at nine evenly spaced SNRs from the lower to the upper edge of its class-specific SNR interval, transcribed by the target Audio LLM, and labeled from the resulting WER. The interval label agrees with the realized-WER class for at least 89.4% of waveforms at every position, so the estimated critical SNRs separate the reliability classes consistently across the whole interval. The predictor is most accurate near the middle of the interval, and its accuracy decreases toward the edges, where waveforms approach the neighboring class.

(b) Filtering and evaluation beyond the filtered population. The fraction of candidate speech-corruption pairs removed by each quality-control criterion is reported in Table 2 (Appendix C) of the original submission:

Audio LLMClean failureReverb. failureSNR gapOrder violationRetained
Qwen2-Audio-7B-Instruct3.8%1.6%7.3%4.0%83.3%
Phi-4-Multimodal-Instruct2.1%1.0%3.8%5.1%88.1%
MOSS-Audio-8B8.5%4.0%3.4%4.4%79.7%

Clean and reverberation failures are recordings that the Audio LLM already transcribes poorly without added interference (WER of at least 5% on the clean or reverberant recording), so their errors cannot be attributed to the acoustic degradation. SNR-gap and order violations remove pairs whose critical SNRs are too close (less than 1 dB apart) or out of order to define stable class-specific intervals. Between 79.7% and 88.1% of candidate pairs are retained, and boundary-adjacent cases are characterized in part (a) (Appendix F).

To characterize the predictor on data that has not passed through this filtering, we evaluated it on real noisy recordings (Appendix H, Figure 9). We use the real recordings of CHiME-3, in which speakers read Wall Street Journal sentences in four noisy environments (bus, cafe, pedestrian area, and street), with speakers, text, microphone, and environments all unseen during training. Each recording is transcribed by Phi-4-Multimodal and labeled directly from its realized WER; none of the quality-control criteria above is applied, and no recording is excluded based on the transcription errors of the Audio LLM. Applied without retraining, the Phi-4-Multimodal predictor reaches 71.37% four-class accuracy and correctly identifies 82.8% of the reliable recordings. The realized WER increases consistently with the predicted class: 1.16%, 4.36%, 18.16%, and 76.71% for recordings predicted as reliable, minor, moderate, and severe degradation.

AuthorsOpenReview ↗

Response to Reviewer 9X8y (Part two)

2. Transfer protocol and Figure 5 (W2, Q2)

(a) Split and number of pairs. The alignment subset is drawn only from the training split of the label banks. It contains 5% of the speech-corruption pairs shared by the source and target label banks: 994 of 19,911 pairs for MOSS from Qwen2, and 1,236 of 24,634 pairs for MOSS from Phi-4.

(b) Disjointness and test set. None of these pairs, speech utterances, or corruption recordings appears in the MOSS-Audio test set, since speech and corruption sources are partitioned across training, validation, and test splits before label construction (Appendix C). Direct transfer, aligned transfer, and training on the MOSS-Audio label bank are all evaluated on the same unchanged MOSS-Audio test set.

(c) Figure 5. The two panels use different threshold sets. The middle panel reports the critical-SNR mismatch of Eq. 6 averaged over all WER thresholds that bound the reliability classes, which gives 2.69 dB (Qwen2/Phi-4) and 3.46 dB (Phi-4/MOSS). The right panel examines estimating the mismatch from subsets of pairs at a single threshold, 10% WER, which gives 2.35 dB and 3.95 dB on the full label bank. Since the alignment estimates a separate shift for each WER threshold (Eq. 7), a single threshold suffices for this analysis. We now state this in Section 4.2 and in the caption of Figure 5.

(d) Variability over 5% subsets. To measure this variability directly, we repeated the aligned transfer with five independently sampled 5% subsets of the shared training pairs. For each subset, we re-estimate the per-boundary shifts (Eq. 7), align the source label bank, retrain the reliability predictor, and evaluate it on the same MOSS-Audio test set. Over the five subsets, the aligned transfer reaches a macro-F1 of 78.97 ± 0.24% for MOSS from Qwen2 and 78.41 ± 0.21% for MOSS from Phi-4.

3. Coverage and error of accepted inputs (Q3)

We added this analysis in Appendix G (Figure 8), using the existing predictions on the fixed in-domain test sets. For each input, we define the predicted probability of degradation as one minus the predicted probability of the reliable class. Accepting inputs whose probability of degradation is below a threshold, and varying the threshold, gives the WER of the accepted inputs at every acceptance rate.

(a) Fixed in-domain test sets. For Qwen2-Audio, the WER of all inputs is 18.60%. Accepting only inputs predicted as reliable lets 35.6% of the inputs proceed without clarification, with a WER of 0.76%; accepting inputs predicted as reliable or minor degradation lets 55.0% proceed with a WER of 4.18%. Phi-4-Multimodal and MOSS-Audio show nearly identical trade-offs: accepting predicted-reliable inputs gives 0.34% and 0.55% WER at 35.7% and 36.5% acceptance, compared with 23.36% and 23.35% without filtering. Since our test sets contain a comparable number of examples from each reliability class by construction, the accepted proportion reflects this construction rather than the class distribution of natural recordings.

(b) Real noisy recordings. On the real CHiME-3 recordings described in point 1(b) (Appendix H, Figures 9 and 10), accepting only recordings predicted as reliable lets 74.4% proceed without clarification, with a WER of 1.16% compared with 2.47% over all recordings; the 25.6% that would trigger a clarification request have a WER of 6.17%. Varying the threshold, the WER of accepted recordings is 1.52% at 90% acceptance, 1.06% at 70%, 0.95% at 50%, and 0.66% at 10%.

Reviewer_UVyZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This work finds that prompting an audio LLM to predict transcription reliability yields poor results. To address this limitation, the authors propose using the audio LLM's audio-encoder representations to predict transcription reliability.

Critical strengths

The paper addresses the important problem of predicting transcription reliability, potentially improving the reliability of audio LLMs in real-world applications. The proposed lightweight predictor operates on frozen audio-encoder representations, making it efficient.

Critical weaknesses

My main concern is the claim that transcription reliability is strongly encoded in audio LLMs' audio-encoder representations. The authors do not investigate whether representations from other pretrained audio encoders, such as HuBERT, WavLM, or Whisper, can achieve comparable performance. This ablation is important for determining whether this reliability information is specific to audio LLMs or is a general property of pretrained audio representations, especially given the paper’s title, “Audio LLMs Know When They Can't Hear You.”

Although the authors fit baseline thresholds using in-domain training data, this approach may not provide sufficient supervision to capture model-specific reliability boundaries. A fairer comparison would involve training a predictor on the audio LLM’s language-model hidden states and comparing the resulting trade-offs.

The paper should emphasize that reliability prediction is performed before transcription generation. Alternatively, the authors should include a self-assessment baseline that provides the audio LLM with its generated transcription and evaluates whether it can recognize its own transcription errors.

Questions for authors

Could you evaluate the proposed predictor using representations from other pretrained audio encoders, such as HuBERT, WavLM, or Whisper?

Could you include a comparable predictor trained on the audio LLM’s language-model hidden states to enable a fairer comparison under similar supervision?

5 comments
AuthorsOpenReview ↗

Response to Reviewer UVyZ

We thank the reviewer for the thoughtful feedback and for recognizing the importance of predicting transcription reliability and the efficiency of our predictor on frozen audio-encoder representations. We have added all three suggested analyses; below we respond point-by-point, referring to the revised manuscript where appropriate.

1. Comparison with other pretrained audio encoders (W1, Q1)

We added this comparison in Appendix K (Table 6). We train the same reliability predictor, with the same learned temporal pooling, classification head, training data, and optimization settings, on six frozen encoders: the audio encoders of the three Audio LLMs and three standalone speech encoders (Whisper-large-v3, WavLM-Large, and HuBERT-Large). Each predictor reads the final layer of its encoder. Each column uses the reliability labels of one target Audio LLM and reports macro-F1 (%) on that Audio LLM's in-domain test set; the best result in each column within each group of encoders is in bold:

Frozen audio encoderQwen2-AudioPhi-4-MMMOSS-Audio
Audio LLM encoders
Qwen2-Audio-7B-Instruct77.5680.0980.79
Phi-4-Multimodal-Instruct74.6084.2979.23
MOSS-Audio-8B74.3180.1080.81
Standalone speech encoders
Whisper-large-v376.7280.7480.82
WavLM-Large70.3575.1375.80
HuBERT-Large69.0773.3674.18

Reliability information is also present in standalone pretrained speech representations, although its predictive strength varies across encoders. WavLM and HuBERT trail the encoder of the target Audio LLM by 5.01 to 10.93 points, while Whisper-large-v3 performs comparably to the Qwen2-Audio and MOSS-Audio encoders (0.84 points lower and 0.01 points higher, respectively) but is 3.55 points below the Phi-4 encoder. The close result for Qwen2-Audio is expected, since its audio encoder is initialized from Whisper-large-v3 and then trained jointly with the language model. The audio encoders of the other Audio LLMs are 0.02 to 4.20 points below the encoder of the target Audio LLM.

The encoder of the target Audio LLM therefore matches or exceeds every alternative, and it is the only one that adds no cost. It already runs for every input, so the reliability decision adds only the lightweight prediction head, and when an input is accepted, the same encoder representations are passed directly to the language model decoder without recomputation. Any alternative encoder must instead be run on every input in addition to the encoder of the target Audio LLM, which is still needed to transcribe the accepted inputs. For example, the Whisper-large-v3 encoder, which shares the architecture of the Qwen2-Audio encoder, requires about 1.89T FLOPs per recording, about 16 times the 0.11T FLOPs of the Phi-4-Multimodal encoder (Table 5). Our predictor reuses the representations of the target encoder for both the reliability decision and the transcription, avoiding this redundant computation. Moreover, the reliability labels themselves remain specific to each Audio LLM, as shown by our label-transfer results (Section 4.2).

AuthorsOpenReview ↗

Response to Reviewer UVyZ (Part 2)

2. Prediction from language-model hidden states (W2, Q2)

We added this comparison for Phi-4-Multimodal in Appendix J (Figure 11). For each recording, we provide the Audio LLM with the recording followed by the same transcription prompt used to construct its reliability labels, with the speech adapter of the language model enabled as during transcription, and run a single forward pass without generating any transcription, so this predictor also operates before transcription. For each of the 32 decoder layers, we average the hidden states over all input tokens (audio tokens and instruction) and train a separate reliability predictor on the result. As a reference, we train the same predictor on the time-averaged final audio-encoder representations. Both use the same classification head, reliability labels, training data, and optimization settings, and are evaluated on the in-domain test set:

RepresentationAccuracyFLOPs per recordingTrainable parameters
Audio encoder (ours)84.31%0.11T0.26M
Best decoder layer (layer 5)84.66%0.29T0.79M
Final decoder layer (layer 32)82.86%1.24T0.79M

Decoder hidden states also contain transcription reliability information. The best decoder layer (layer 5) improves accuracy over the audio encoder by a modest 0.35 points (84.66% vs. 84.31%), but obtaining its representation requires about 2.6x the computation (0.29T vs. 0.11T FLOPs), since reading a decoder layer requires running the audio encoder and all preceding decoder layers over the full input. Deeper decoder layers provide no consistent improvement, with the final layer decreasing to 82.86% accuracy while its cost rises to 1.24T FLOPs (10.9x). The decoder-based predictor also needs about three times more trainable parameters, because the decoder hidden states are higher-dimensional. Thus, accessing decoder representations offers little additional predictive benefit relative to the extra computation.

3. Prediction before generation and transcript-conditioned self-assessment (W3)

(a) Prediction before generation. Our predictor operates only on the frozen audio-encoder representations and makes its decision before the language model decoder runs, so no transcription is generated for rejected inputs. This is stated in the abstract ("predicts the reliability class before generation", line 21), in the introduction, which motivates the audio encoder as the representation computed "before language generation" (line 53) and proposes predicting reliability from it "rather than asking the decoder whether it understood the audio" (lines 73-74), in Section 2 ("without generating a transcript", line 153), in Section 3.2, which notes that the audio encoder is already executed to process the input (lines 283-284), in Appendix I ("the reliability decision can be made without executing the language model decoder", lines 1063-1064), in Appendix J ("makes its decision before the language model decoder is executed", lines 1120-1121), and in Appendix K (lines 1210-1212).

(b) Transcript-conditioned self-assessment. We added the suggested self-assessment baseline (Figure 1, Appendix B). In the original zero-shot and two-shot settings, Qwen2-Audio is asked, before transcribing, whether it can transcribe a recording reliably. In the new setting, it receives the recording together with its own transcription of that recording, the same transcription used to compute its WER, and is asked whether this transcription is reliable. Seeing its own transcription does not help: the model becomes even more overconfident than in the zero-shot and two-shot settings, judging 92% of its transcriptions with WER above 30% as reliable. The Audio LLM therefore cannot detect its own transcription errors even after generation, which further motivates a dedicated reliability predictor that operates before generation.

AuthorsOpenReview ↗

Follow-up on the rebuttal

Dear Reviewer UVyZ,

As about 12 hours remain in the discussion period, we would like to follow up on our response. We have tried to address all of your comments and suggestions in the revised manuscript:

  • Comparison with other pretrained audio encoders: we train the same reliability predictor on six frozen encoders, including Whisper-large-v3, WavLM-Large, and HuBERT-Large (Appendix K, Table 6).
  • Prediction from language-model hidden states: we compare audio-encoder representations with every decoder layer of the Audio LLM (Appendix J, Figure 11).
  • Transcript-conditioned self-assessment: we add a baseline in which the Audio LLM judges its own transcription (Figure 1, Appendix B).

We would greatly appreciate it if you could let us know whether these revisions address your concerns, and we would love to hear your thoughts on them.

Thank you again for your time and constructive feedback.

Reviewer_AHdrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper predicts whether an Audio LLM can reliably transcribe an input recording before generating a transcript. The authors construct four-class reliability labels by searching for model-specific SNR boundaries at different WER thresholds. They train a small pooling module and classifier on a frozen audio encoder. Experiments compare reliability prediction methods, test cross-domain generalization, and study label transfer across three Audio LLMs.

Critical strengths
  1. Predicting transcription failures before decoding is a useful goal. The predictor is lightweight and does not modify the underlying Audio LLM.
  2. The results show clear gains over the evaluated baselines. Cross-domain evaluation and cross-model label transfer provide useful additional evidence.
Critical weaknesses
  1. The comparisons do not isolate the value of the target model’s encoder. Most baselines use a single score with fitted thresholds, while the proposed method trains a pooling module and MLP on high-dimensional features. Although Fe-WER is also trained, a matched comparison using other frozen speech encoders with the same classification head is missing.
  2. Generalization is tested under a limited set of conditions. Both evaluation sets use controlled corruption and the same label-construction procedure. Filtering also removes samples that are already transcribed poorly in clean conditions. Performance on naturally degraded recordings and other causes of transcription failure remains unclear.
  3. The practical benefit is not directly evaluated. The experiments measure classification performance, but do not show whether accepting or rejecting queries based on these predictions improves downstream reliability. Reporting transcription error at different acceptance rates would help.
  4. (Minor) The scope is limited to transcription reliability measured by WER. It remains unclear whether the predictor can detect failures in audio question answering or reasoning. For example, when LALM cannot reason correctly about an audio QA, can we use a similar method to detect whether it's because the degradation in audio or not.
Questions for authors
  1. How do other frozen speech encoders perform with the same pooling module, classifier, training data, and optimization settings?
  2. Could you evaluate the predictor on a real-world ASR test set and report the actual WER of recordings predicted as reliable, together with the proportion of recordings accepted? This would directly test whether the predictor can identify trustworthy transcriptions and thus enhance the reliability of the asr ability of an LALM.
4 comments
AuthorsOpenReview ↗

Response to Reviewer AHdr

We thank the reviewer for the constructive feedback and for recognizing the usefulness and lightweight design of our approach. Below we respond point-by-point and summarize the corresponding updates in the revised manuscript.

1. Comparison with other frozen speech encoders (W1, Q1)

We added this comparison in Appendix K (Table 6). We train the same reliability predictor, with the same learned temporal pooling, classification head, training data, and optimization settings, on six frozen encoders: the audio encoders of the three Audio LLMs and three standalone speech encoders (Whisper-large-v3, WavLM-Large, and HuBERT-Large). Each predictor reads the final layer of its encoder. Each column uses the reliability labels of one target Audio LLM and reports macro-F1 (%) on that Audio LLM's in-domain test set; the best result in each column within each group of encoders is in bold:

Frozen audio encoderQwen2-AudioPhi-4-MMMOSS-Audio
Audio LLM encoders
Qwen2-Audio-7B-Instruct77.5680.0980.79
Phi-4-Multimodal-Instruct74.6084.2979.23
MOSS-Audio-8B74.3180.1080.81
Standalone speech encoders
Whisper-large-v376.7280.7480.82
WavLM-Large70.3575.1375.80
HuBERT-Large69.0773.3674.18

The audio encoders of the other Audio LLMs do not improve over the encoder of the target Audio LLM, scoring 0.02 to 4.20 points lower. WavLM and HuBERT trail the target encoder by 5.01 to 10.93 points, suggesting that self-supervised representations capture less model-specific reliability information. Whisper-large-v3 performs comparably to the Qwen2-Audio and MOSS-Audio encoders (0.84 points lower and 0.01 points higher, respectively) but is 3.55 points below the Phi-4 encoder. The close result for Qwen2-Audio is expected, since its audio encoder is initialized from Whisper-large-v3 and then trained jointly with the language model.

The encoder of the target Audio LLM therefore matches or exceeds every alternative, and it is the only one that adds no cost. It already runs for every input, so the reliability decision adds only the 0.26M to 0.33M parameter prediction head, and when an input is accepted, the same encoder representations are passed directly to the language model decoder without recomputation. Any alternative encoder must instead be run on every input in addition to the encoder of the target Audio LLM, which is still needed to transcribe the accepted inputs. For example, the Whisper-large-v3 encoder, which shares the architecture of the Qwen2-Audio encoder, requires about 1.89T FLOPs per recording, about 16 times the 0.11T FLOPs of the Phi-4-Multimodal encoder (Table 5). Our predictor reuses the representations of the target encoder for both the reliability decision and the transcription, avoiding this redundant computation.

2. Evaluation on naturally degraded recordings (W2, Q2)

We added this evaluation in Appendix H (Figure 9). We use the real recordings of CHiME-3, in which speakers read Wall Street Journal sentences in four noisy environments (bus, cafe, pedestrian area, and street), captured by a tablet microphone. The speakers, text domain, microphone, and acoustic environments are all unseen during training. We apply the predictor trained on the in-domain label bank of Phi-4-Multimodal, without any retraining or recalibration.

Each recording is labeled directly from the WER of Phi-4-Multimodal on it, using the same prompt and decoding settings as in our label construction. No recording is excluded based on the transcription errors of the Audio LLM.

Predicted classRecordingsActual WER
Reliable74.4%1.16%
Minor degradation22.9%4.36%
Moderate degradation2.5%18.16%
Severe degradation0.2%76.71%
All recordings100%2.47%

The actual WER increases consistently with the predicted class. Accepting only the recordings predicted as reliable lets 74.4% of recordings proceed with a WER of 1.16%, compared with 2.47% over all recordings. The remaining 25.6%, which would trigger a clarification request, have a WER of 6.17%. The predictor therefore halves the transcription error of accepted queries while letting three quarters of them proceed. The predictor reaches 71.37% four-class accuracy and correctly identifies 82.8% of the reliable recordings. Unlike our label banks, whose test sets contain examples of every reliability class, the class distribution of real recordings is set by the robustness of the Audio LLM: Phi-4-Multimodal transcribes most CHiME-3 recordings without error, so 76.1% are reliable and only 0.9% are severely degraded.

AuthorsOpenReview ↗

Response to Reviewer AHdr (Part 2)

3. Transcription error at different acceptance rates (W3)

To further demonstrate the practical benefit of the predictor, we added a coverage/error analysis for both real recordings (Appendix H, Figure 10) and the fixed in-domain test sets (Appendix G, Figure 8). For each input, we define the predicted probability of degradation as one minus the predicted probability of the reliable class. An application accepts the inputs whose probability of degradation is below a threshold and asks the user to repeat the others; varying this threshold traces the WER of the accepted inputs against the proportion of inputs that are accepted.

(a) Real noisy recordings. Figure 10 shows this trade-off on the real CHiME-3 recordings. The WER of the accepted recordings decreases steadily as the acceptance criterion becomes stricter, from 2.47% when all recordings are accepted to 0.66% when only 10% are accepted:

Accepted recordings10%30%50%70%80%90%100%
WER of accepted0.66%0.79%0.95%1.06%1.24%1.52%2.47%

(b) Fixed in-domain test sets. Figure 8 shows the same analysis on the fixed in-domain test set of Qwen2-Audio, whose WER over all inputs is 18.60%. The WER of the accepted inputs stays close to zero until about a third of the inputs are accepted, and then rises gradually as less reliable inputs are admitted.

Accepted inputs10%30%50%70%80%90%100%
WER of accepted0.20%0.25%3.16%7.12%9.24%12.09%18.60%

When the predicted class is used directly as the decision, accepting only inputs predicted as reliable lets 35.6% of the inputs proceed with a WER of 0.76%, and additionally accepting inputs predicted as minor degradation lets 55.0% proceed with a WER of 4.18%. For Phi-4-Multimodal and MOSS-Audio, accepting only inputs predicted as reliable lets 35.7% and 36.5% of the inputs proceed with a WER of 0.34% and 0.55%, compared with 23.36% and 23.35% when all inputs are accepted.

In both settings, rejecting the inputs that the predictor flags as degraded substantially lowers the transcription error of the accepted inputs, and the threshold lets an application choose its own balance between accepting more inputs and keeping their error low.

4. Quality-control filtering of the label banks (W2)

The filtering removes only a small fraction of the data. Table 2 (Appendix C) of the original submission lists the fraction of candidate speech-corruption pairs removed by each quality-control criterion:

Audio LLMClean failureReverb. failureSNR gapOrder violationRetained
Qwen2-Audio-7B-Instruct3.8%1.6%7.3%4.0%83.3%
Phi-4-Multimodal-Instruct2.1%1.0%3.8%5.1%88.1%
MOSS-Audio-8B8.5%4.0%3.4%4.4%79.7%

Recordings that the Audio LLM already transcribes poorly in clean conditions account for only 2.1% to 8.5% of the candidate pairs, and 79.7% to 88.1% of all pairs are retained. We remove these recordings so that each reliability label reflects transcription errors caused by the acoustic degradation, rather than errors the Audio LLM already makes on the clean speech.

5. Extension beyond transcription

We agree that this is an important extension. In this work, we focus on transcription reliability, where the WER of each recording provides an objective, model-specific label. Extending the predictor to detect when degraded audio leads the Audio LLM to hallucinate or answer incorrectly in audio question answering and reasoning is a natural next step that we leave to future work.

Thank you for the comprehensive feedback. As most of my concerns have been addressed, I have raised my scores accordingly. I believe that these additional comparisons and evaluations have greatly enhanced the paper’s depth.

AuthorsOpenReview ↗

Thank you for the follow-up

We thank the reviewer for the follow-up and for raising the score. We are glad that the additional comparisons and evaluations addressed most of the concerns. The comparison with other frozen speech encoders (Appendix K), the evaluation on the real recordings of CHiME-3 (Appendix H), and the coverage analysis (Appendices G and H) have improved the paper, and we thank the reviewer again for the detailed review and recommendations.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General comment to all reviewers

We thank all reviewers for their constructive feedback, and for recognizing the practical value of predicting transcription reliability before decoding (AHdr, UVyZ, 9X8y), the lightweight predictor that leaves the Audio LLM unchanged (AHdr, UVyZ, 9X8y), the consistent gains across three model families and both evaluation settings (AHdr, 9X8y), and the label-bank transfer study (AHdr, 9X8y).

We have uploaded a revised manuscript, in which all revisions are highlighted in blue. The main revisions are:

(i) Comparison with other representations. We train the same predictor on six frozen audio encoders, including Whisper-large-v3, WavLM-Large, and HuBERT-Large (Appendix K, Table 6), and on every decoder layer of Phi-4-Multimodal (Appendix J, Figure 11).

(ii) Evaluation on real noisy recordings. We apply the predictor, without retraining, to the real recordings of CHiME-3 and report the WER and proportion of accepted recordings (Appendix H, Figures 9 and 10).

(iii) Transcript-conditioned self-assessment. We add a baseline in which the Audio LLM judges its own transcription (Figure 1, Appendix B).

(iv) Evaluation across the class-specific SNR intervals. We evaluate the predictor at nine SNRs from the lower to the upper edge of each class-specific SNR interval (Appendix F, Figure 7).

(v) Coverage analysis and transfer protocol. We report the coverage and error of accepted inputs on the fixed test sets (Appendix G, Figure 8) and clarify the label-transfer protocol (Section 4.2).

AuthorsOpenReview ↗

Summary of the discussion

Dear Area Chair and Reviewers,

We thank all reviewers for their constructive feedback and engagement during the discussion period. All three reviewers (AHdr, UVyZ, and 9X8y) have indicated that their concerns were addressed and have raised their ratings.

The main revisions, highlighted in blue in the revised manuscript, are:

  • Comparison with other representations: the same reliability predictor trained on six frozen audio encoders (Appendix K) and on every decoder layer of the Audio LLM (Appendix J).
  • Evaluation against realized WER: all results rescored with labels assigned from the realized WER of the target Audio LLM, with our predictor remaining the best method on every metric (Section 3.1), and an evaluation across the class-specific SNR intervals (Appendix F).
  • Real noisy recordings and coverage: evaluation on the real recordings of CHiME-3 without retraining (Appendix H), and the proportion and error of accepted inputs (Appendices G and H).
  • Transcript-conditioned self-assessment: a baseline in which the Audio LLM judges its own transcription (Figure 1, Appendix B).
  • Transfer protocol: the alignment subset sizes and the variability over five independently sampled subsets (Section 4.2, Appendix M).

We thank the Area Chair and the reviewers again for their time and effort.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.