- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether an Audio LLM's transcription reliability can be predicted before language-model decoding. It constructs model-specific label banks by searching for SNR values at which transcription WER crosses 0%, 10%, and 30%, then trains learned temporal pooling and a small classifier over a frozen audio encoder to predict four reliability classes. On Qwen2-Audio, the reported macro-F1 is 81.10% in-domain and 78.09% cross-domain, with additional results on Phi-4-Multimodal and MOSS-Audio. The paper also examines prompted self-assessment, encoder-depth and pooling ablations, and reuse of another model's label bank. Boundary shifts estimated from 5% of shared speech-corruption pairs substantially improve the two reported transfers to MOSS. The intended application is to identify potentially unreliable voice inputs before responding.
Critical strengths
-
A useful problem with a simple, potentially practical solution. Predicting transcription reliability before decoding could support early clarification decisions without changing the underlying Audio LLM. The reported improvements are consistent across three model families and both evaluation settings (Tables 1, 3, and 4). For Qwen2-Audio, the macro-F1 improvements over the strongest baseline are 10.33 and 11.93 points. Appendix C also describes disjoint speech speakers/utterances, noise recordings, and room environments across the in-domain splits. These are meaningful positive results for a lightweight prediction head, although the deployment implications still depend on the validity and scope of the labels.
-
The label-bank reuse study adds value beyond a standard encoder probe. The authors investigate how model-dependent corruption boundaries affect transfer and propose a simple correction using limited target-model measurements. Table 7 reports that per-boundary median alignment reduces the accuracy deficit for MOSS <- Qwen2 from 9.69 to 1.70 points and for MOSS <- Phi-4 from 12.86 to 2.05 points. If confirmed under a clearly separated calibration/evaluation protocol, this would make the approach substantially less expensive to adapt to another Audio LLM.
Critical weaknesses
-
The validity of the per-example evaluation labels is not sufficiently established. Section 3.1 and Appendix C estimate three critical SNRs with binary search under an approximate monotonicity assumption, then sample training and fixed evaluation waveforms from the resulting intervals. The description does not explicitly establish whether every fixed validation/test waveform is transcribed again and labeled using its realized WER. Ordered boundaries and margins around them do not by themselves establish that WER stays within the assigned class throughout each interval. If test labels are inherited from those intervals, the reported metrics could partly measure agreement with the label-construction approximation rather than actual transcription reliability. This matters directly to the central claim. In addition, the evaluation follows filtering that removes clean/reverberation failures and ambiguous boundary cases, so the conclusions should be scoped to this selected population until performance on excluded or boundary-adjacent cases is characterized. I would like a direct audit against realized WER before treating the reported gains as convincing evidence of failure detection.
-
The cross-model transfer evidence needs a clearer and internally consistent protocol. Section 4.2 does not specify clearly which split supplies the 5% target-labeled pairs, whether these pairs are disjoint from the reported evaluation set, and whether all transfer/alignment variants use the same fixed target test population after filtering. This is an ambiguity, not evidence that leakage occurred. There is also an apparent discrepancy within Figure 5: the middle panel places the Qwen2/Phi-4 mismatch at approximately 2.7 dB and the Phi-4/MOSS mismatch at approximately 3.45 dB, whereas the right panel's full-pair estimates are approximately 2.35 and 3.95 dB, respectively. If these use different pair sets or definitions, that needs to be stated; otherwise the figures should agree under Eq. (6). Because boundary mismatch and low-cost alignment form a central contribution, these details materially affect my assessment.
Overall, the method and reported gains are promising, but these two unresolved validation issues lead me to weak rejection at present. A convincing realized-WER audit and clarification of the transfer protocol could change my recommendation.
Post-rebuttal update (19 Nov 2026):
The revised evaluation against realized WER, the clarified transfer protocol and the coverage/WER analyses address my main concerns. The latest revision also includes the calibration subset sizes and the five-run variability results requested in the discussion. I am keeping my rating at 3 (weak acceptance). The evidence supports transcription reliability; broader audio-understanding failure detection remains outside the demonstrated scope. This is my final recommendation after the rebuttal, and I have no further questions.
Questions for authors
-
Clarification and validation of the existing results: Are the labels used to score Tables 1, 3, and 4 computed from a fresh transcription of each fixed evaluation waveform, or inherited from its sampled SNR interval? Please report the agreement between interval labels and realized-WER classes on an independently checked subset (including samples near the retained interval edges), and whether the main comparison changes when scored against realized WER.
-
Clarification of the existing transfer experiment: Which split and how many shared pairs supply the 5% alignment data? Are they disjoint from evaluation, and is one unchanged target-model test set used for direct, aligned, and target-label-bank training? Please also reconcile the middle/right-panel mismatch values in Figure 5 and report variability of the aligned-transfer results over independently sampled 5% subsets.
-
Scope of the proposed clarification use case: On the existing fixed tests, what fraction of inputs would proceed without clarification, and what is the realized transcription-error rate among those accepted inputs at a specified operating point? Figure 6 provides useful class-conditional error rates, but a coverage/error trade-off would clarify whether the predictor supports the proposed decision. This is a request to analyze the existing predictions, not a request for a new user study.
5 comments
Response to Reviewer 9X8y
We are grateful to the reviewer for the detailed and constructive review, for recognizing the practical value of our approach, its consistent gains across three model families, and the contribution of the label-bank transfer study, and for identifying precisely which validations would make the evidence convincing. Below we respond point-by-point and summarize the corresponding updates in the revised manuscript.
1. Validity of the evaluation labels and scope of the evaluation (W1, Q1)
(a) Audit against realized WER. We thank the reviewer for this important point. Every fixed test waveform was transcribed by the target Audio LLM, and its WER was stored. However, the labels used to score the tables of the original submission were inherited from the SNR interval from which each test waveform was generated. We audited the agreement between this interval label and the reliability class of the realized WER of each test waveform:
| Audio LLM | In-domain test | Cross-domain test |
|---|---|---|
| Qwen2-Audio-7B-Instruct | 94.8% | 93.6% |
| Phi-4-Multimodal-Instruct | 95.9% | 95.0% |
| MOSS-Audio-8B | 96.9% | 97.1% |
The two labels agree for 93.6% to 97.1% of test waveforms, and 92.9% of the disagreements are between neighboring reliability classes.
Following the reviewer's suggestion, the label of every test example in the revised manuscript is assigned from the realized WER of the target Audio LLM on that example (Section 3.1), and all results are rescored with the same predictions and the same fitted baseline thresholds: the main comparison (Tables 1, 3, and 4), the confusion matrices (Figure 6), the label-transfer results (Figure 5), and the ablations (Appendix L). The analyses added in the revision, including the encoder comparison (Table 6), the evaluation across the class-specific SNR intervals (Figure 7), the coverage analysis (Figure 8), and the decoder-layer comparison (Figure 11), are likewise evaluated with labels assigned from the realized WER. The changes in the results are minimal, and our predictor remains the best method on every metric for all three Audio LLMs in both evaluation settings.
To characterize the labels and the predictor near the interval edges, we added an evaluation in Appendix F (Figure 7). Each in-domain test example is generated at nine evenly spaced SNRs from the lower to the upper edge of its class-specific SNR interval, transcribed by the target Audio LLM, and labeled from the resulting WER. The interval label agrees with the realized-WER class for at least 89.4% of waveforms at every position, so the estimated critical SNRs separate the reliability classes consistently across the whole interval. The predictor is most accurate near the middle of the interval, and its accuracy decreases toward the edges, where waveforms approach the neighboring class.
(b) Filtering and evaluation beyond the filtered population. The fraction of candidate speech-corruption pairs removed by each quality-control criterion is reported in Table 2 (Appendix C) of the original submission:
| Audio LLM | Clean failure | Reverb. failure | SNR gap | Order violation | Retained |
|---|---|---|---|---|---|
| Qwen2-Audio-7B-Instruct | 3.8% | 1.6% | 7.3% | 4.0% | 83.3% |
| Phi-4-Multimodal-Instruct | 2.1% | 1.0% | 3.8% | 5.1% | 88.1% |
| MOSS-Audio-8B | 8.5% | 4.0% | 3.4% | 4.4% | 79.7% |
Clean and reverberation failures are recordings that the Audio LLM already transcribes poorly without added interference (WER of at least 5% on the clean or reverberant recording), so their errors cannot be attributed to the acoustic degradation. SNR-gap and order violations remove pairs whose critical SNRs are too close (less than 1 dB apart) or out of order to define stable class-specific intervals. Between 79.7% and 88.1% of candidate pairs are retained, and boundary-adjacent cases are characterized in part (a) (Appendix F).
To characterize the predictor on data that has not passed through this filtering, we evaluated it on real noisy recordings (Appendix H, Figure 9). We use the real recordings of CHiME-3, in which speakers read Wall Street Journal sentences in four noisy environments (bus, cafe, pedestrian area, and street), with speakers, text, microphone, and environments all unseen during training. Each recording is transcribed by Phi-4-Multimodal and labeled directly from its realized WER; none of the quality-control criteria above is applied, and no recording is excluded based on the transcription errors of the Audio LLM. Applied without retraining, the Phi-4-Multimodal predictor reaches 71.37% four-class accuracy and correctly identifies 82.8% of the reliable recordings. The realized WER increases consistently with the predicted class: 1.16%, 4.36%, 18.16%, and 76.71% for recordings predicted as reliable, minor, moderate, and severe degradation.
Response to Reviewer 9X8y (Part two)
2. Transfer protocol and Figure 5 (W2, Q2)
(a) Split and number of pairs. The alignment subset is drawn only from the training split of the label banks. It contains 5% of the speech-corruption pairs shared by the source and target label banks: 994 of 19,911 pairs for MOSS from Qwen2, and 1,236 of 24,634 pairs for MOSS from Phi-4.
(b) Disjointness and test set. None of these pairs, speech utterances, or corruption recordings appears in the MOSS-Audio test set, since speech and corruption sources are partitioned across training, validation, and test splits before label construction (Appendix C). Direct transfer, aligned transfer, and training on the MOSS-Audio label bank are all evaluated on the same unchanged MOSS-Audio test set.
(c) Figure 5. The two panels use different threshold sets. The middle panel reports the critical-SNR mismatch of Eq. 6 averaged over all WER thresholds that bound the reliability classes, which gives 2.69 dB (Qwen2/Phi-4) and 3.46 dB (Phi-4/MOSS). The right panel examines estimating the mismatch from subsets of pairs at a single threshold, 10% WER, which gives 2.35 dB and 3.95 dB on the full label bank. Since the alignment estimates a separate shift for each WER threshold (Eq. 7), a single threshold suffices for this analysis. We now state this in Section 4.2 and in the caption of Figure 5.
(d) Variability over 5% subsets. To measure this variability directly, we repeated the aligned transfer with five independently sampled 5% subsets of the shared training pairs. For each subset, we re-estimate the per-boundary shifts (Eq. 7), align the source label bank, retrain the reliability predictor, and evaluate it on the same MOSS-Audio test set. Over the five subsets, the aligned transfer reaches a macro-F1 of 78.97 ± 0.24% for MOSS from Qwen2 and 78.41 ± 0.21% for MOSS from Phi-4.
3. Coverage and error of accepted inputs (Q3)
We added this analysis in Appendix G (Figure 8), using the existing predictions on the fixed in-domain test sets. For each input, we define the predicted probability of degradation as one minus the predicted probability of the reliable class. Accepting inputs whose probability of degradation is below a threshold, and varying the threshold, gives the WER of the accepted inputs at every acceptance rate.
(a) Fixed in-domain test sets. For Qwen2-Audio, the WER of all inputs is 18.60%. Accepting only inputs predicted as reliable lets 35.6% of the inputs proceed without clarification, with a WER of 0.76%; accepting inputs predicted as reliable or minor degradation lets 55.0% proceed with a WER of 4.18%. Phi-4-Multimodal and MOSS-Audio show nearly identical trade-offs: accepting predicted-reliable inputs gives 0.34% and 0.55% WER at 35.7% and 36.5% acceptance, compared with 23.36% and 23.35% without filtering. Since our test sets contain a comparable number of examples from each reliability class by construction, the accepted proportion reflects this construction rather than the class distribution of natural recordings.
(b) Real noisy recordings. On the real CHiME-3 recordings described in point 1(b) (Appendix H, Figures 9 and 10), accepting only recordings predicted as reliable lets 74.4% proceed without clarification, with a WER of 1.16% compared with 2.47% over all recordings; the 25.6% that would trigger a clarification request have a WER of 6.17%. Varying the threshold, the WER of accepted recordings is 1.52% at 90% acceptance, 1.06% at 70%, 0.95% at 50%, and 0.66% at 10%.
Note on AI tool use during my review
Following the PCs' request, I'm adding a note about how AI tools were used during my reviewing process. I discussed this paper with ChatGPT (GPT-6-Astra) while working on my review. I used it as a sounding board: I shared my own reading notes — mainly the label-validity concern about the SNR-interval label construction in Sec 3.1 / Appendix C, and the protocol ambiguities in the cross-model transfer experiment (Sec 4.2, Figure 5) — and asked it to check whether my criticisms were technically sound and whether I had overlooked obvious counterarguments. It also helped me polish the wording of the review text. My assessment of the paper, the rating, and all specific claims in the review are my own: I verified every cited detail (the label construction procedure, Table 7, the Figure 5 mismatch values, etc.) directly against the manuscript. The AI discussion helped me articulate the label-validity concern more precisely, but it did not change my overall assessment. My questions for the authors in the review stand as written, and I look forward to their responses.