- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AuditEar, a speech quality evaluation LLM fine-tuned from Qwen3-Omni-30B-A3B-Instruct using the proposed ProQA dataset. ProQA consists of 634 speech recordings annotated across 18 dimensions of speech quality. The authors introduce a multi-stage training pipeline involving SFT, RGSD+OPSD, and GRPO. Experimental results demonstrate promising in-domain performance and cross-domain generalization on the QualiSpeech and SpeechEval benchmarks.
Critical strengths
The training pipeline is carefully designed, incorporating multiple training stages to further exploit the potential of speech LLMs for speech quality evaluation. The combination of different training strategies provides a systematic approach to improving the model's evaluation capabilities and generalization.
Critical weaknesses
-
Limited dataset scale and contribution. Although the authors acknowledge the relatively small scale of ProQA and emphasize the difficulty of collecting human annotations, the dataset size remains a significant concern. For comparison, the existing QualiSpeech dataset contains more than 10,000 speech samples and over 70,000 human annotations, substantially exceeding the scale of ProQA. Given that considerably larger speech quality evaluation datasets already exist, the incremental contribution of ProQA appears somewhat limited.
-
Limited cross-domain generalization and practical applicability. Despite the improvements achieved through the proposed multi-stage training pipeline, AuditEar still underperforms models fine-tuned on the target domains when evaluated on out-of-domain datasets such as QualiSpeech and SpeechEval. This suggests that the limited scale of the training data may constrain the model's generalization capabilities. In practical scenarios involving diverse and previously unseen speech samples, AuditEar may therefore struggle to outperform models trained on larger speech quality evaluation datasets. This limitation raises concerns about the practical applicability of AuditEar as a general-purpose speech quality evaluator.
Questions for authors
-
Could the authors provide additional comparisons with SSL-based speech quality assessment methods, such as UTMOS [1], particularly on rating-based evaluation tasks? Such comparisons would help establish the effectiveness of AuditEar relative to established speech quality prediction approaches.
-
According to Table 3, the SFT model always underperforms the base Qwen3-Omni model. Could the authors provide further insights into this observation? In particular, why does supervised fine-tuning lead to a performance degradation despite using task-specific speech quality evaluation data?
[1] Saeki, Takaaki, et al. "Utmos: Utokyo-sarulab system for voicemos challenge 2022." arXiv preprint arXiv:2204.02152 (2022).
6 comments
Author Response to Reviewer uB3K (2/3)
W2. Cross-domain generalization and practical applicability
We would like to highlight that the models the reviewer refers to are fine-tuned on the QualiSpeech and SpeechEval datasets themselves.
- In-domain vs. out-of-domain. By out-of-domain generalization, we mean that AuditEar is never trained on QualiSpeech or SpeechEval data. It is trained only on the ProQA training split, which differs substantially from the training splits of both benchmarks, as described in our response to W1. ProQA covers different quality axes (such as mic quality and placement, flutter echoes and sibilance), different tasks (per-axis severity, issue tags, written assessments and recommended fixes), and different question formats. QualiSpeech and SpeechEval instead use different axes and task formats from ProQA. In contrast, QualiSpeech-SALMONN is fine-tuned on QualiSpeech's training data and SQ-LLM is fine-tuned on SpeechEval's data, so their scores on their own benchmarks are in-domain results.
- Comparison with zero-shot models. The fairer comparison is with models that have also not been trained on these out-of-domain benchmarks. AuditEar performs much better on the QualiSpeech benchmark than other zero-shot audio-language models evaluated in the QualiSpeech paper, such as Qwen-Audio, Qwen2-Audio and SALMONN, whose best results we list in Table 5. AuditEar also improves on its base model, Qwen3-Omni, from 0.169 to 0.628 on QualiSpeech and from 0.564 to 0.743 on SpeechEval, as shown in the table below. Prior work also treats generalization to unseen domains as a separate and harder setting, in which quality predictors typically fall below models trained on the target data [3]. Since our goal was to test generalization to out-of-domain datasets, we did not add any of QualiSpeech's or SpeechEval's data to AuditEar's training.
We further compared AuditEar with these two audio-quality assessment judges, QualiSpeech-SALMONN and SQ-LLM, on all three benchmarks, each evaluated on the same clips with its own prompt format (Appendix A.10).
| Model | Trained on | QualiSpeech (Overall Quality) | SpeechEval (Pairwise Comparison) | ProQA (Pairwise Comparison Score) |
|---|---|---|---|---|
| Qwen3-Omni | – | 0.169 | 0.564 | 0.574 |
| QualiSpeech SALMONN-7B [1] | QualiSpeech | 0.712 | 0.696 | 0.689 |
| SQ-LLM [2] | SpeechEval | 0.589 | 0.793 | |
| AuditEar | ProQA | 0.886 |
- Trained only on ProQA, AuditEar ranks second on both out-of-domain benchmarks, behind only the model fine-tuned on that benchmark's own data. AuditEar scores 0.628 on QualiSpeech and 0.743 on SpeechEval. AuditEar also increases the scores of its base model, Qwen3-Omni, from 0.169 to 0.628 on QualiSpeech and from 0.564 to 0.743 on SpeechEval, which shows the value of ProQA's expert annotations.
- Although QualiSpeech-SALMONN and SQ-LLM are trained on more audio and annotations than AuditEar, each of them comes third on the other external benchmark. QualiSpeech-SALMONN scores 0.696 on SpeechEval, against 0.743 for AuditEar, and SQ-LLM scores 0.589 on QualiSpeech, against 0.628 for AuditEar. On ProQA they reach only 0.689 and 0.713, against 0.886 for AuditEar. Since Qwen3-Omni without ProQA training scores lowest in every column, the backbone alone does not explain this, which highlights the importance of the high-quality expert annotations in ProQA.
- On SpeechEval, AuditEar matches SQ-LLM on English, and the remaining gap comes from other languages. SpeechEval covers English, French, Japanese and Chinese, and SQ-LLM is trained on all four languages, whereas ProQA focuses on English speech. On the same SpeechEval test pairs split by language, the results are:
| Model | English | French | Japanese | Chinese |
|---|---|---|---|---|
| Qwen3-Omni | 0.587 | 0.603 | 0.525 | 0.541 |
| SQ-LLM | 0.826 | 0.778 | 0.832 | |
| AuditEar | 0.736 |
On English, AuditEar slightly outperforms SQ-LLM without any SpeechEval training (0.736 vs. 0.729). The gap is in French, Japanese and Chinese, yet even there AuditEar improves on its base model by 0.18–0.21. We have added the cross-benchmark and per-language results to the paper (Appendix A.10).
References
[1] Wang et al. "QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptions." ACL 2025.
[2] Wang et al. "SpeechLLM-as-Judges: Towards General and Interpretable Speech Quality Evaluation." ACL 2026.
[3] Cooper et al. "Generalization Ability of MOS Prediction Networks." ICASSP 2022.
Author Response to Reviewer uB3K (3/3)
Q1. Comparison with UTMOS and other quality predictors
We have evaluated UTMOS [4] and DNSMOS [5, 6] on ProQA.
- UTMOS [4] predicts a mean opinion score from self-supervised speech features, and DNSMOS predicts MOS-style scores for overall speech quality (P.808 [5]) and for signal, background and overall quality (P.835 [6]). Both output numerical scores for each recording.
- Due to these models' limitations, such as producing only numerical scores and no per-axis severity ratings, issue tags, written assessments or recommended fixes, they cannot be fully evaluated on all the tasks of our benchmark.
We therefore evaluated them on a subset of ProQA on pairwise comparison, taking the recording with the higher predicted score as the better one. AuditEar scores higher than UTMOS and DNSMOS. In addition to choosing the better recording in each pair, AuditEar also explains why it is better, and identifies the issues and recommends fixes for all axes, which these predictors cannot do. We have added this comparison to the paper (Appendix A.11).
| Model | Pairwise Comparison Score |
|---|---|
| AuditEar | 0.886 |
| UTMOS [4] | 0.718 |
| DNSMOS P.808 [5] | 0.713 |
| DNSMOS P.835 overall [6] |
Q2. Stage-1 SFT vs. the base model
Across Tables 3 and 4, Stage-1 SFT actually outperforms the base Qwen3-Omni model on the discriminative tasks (severity accuracy, issue detection, pairwise accuracy and position consistency), but it underperforms it on the free-text metrics (issue-, tag- and fix-F1). This can be explained as follows:
- To evaluate the free-text metrics, an LLM judge first extracts keywords from the model response and the ground truth answer and then matches them. The base model writes long, generic responses, as shown by the fact that, on average, its responses list 3.6 issues against 2.5 in the ground truth. Many of its items do not match, so its precision is low, but because it lists so many keywords, enough items match to give it high recall.
- Stage-1 SFT writes short, specific answers that are usually correct, so its precision is high. However, it under-reports some errors, so its recall is low. We have added this precision/recall analysis to the paper (Appendix A.12).
The precision and recall scores of the base Qwen3-Omni and Stage-1 SFT are shown below:
| Metric | Stage-1 SFT Precision | Stage-1 SFT Recall | Qwen3-Omni Precision | Qwen3-Omni Recall |
|---|---|---|---|---|
| Single-Axis · issue | 0.59 | 0.21 | 0.41 | 0.60 |
| Single-Axis · tag | 0.60 | 0.20 | 0.33 | 0.52 |
| Single-Axis · fix | 0.45 | 0.13 | 0.34 | 0.54 |
| Holistic · issue | 0.60 | 0.19 | 0.54 | 0.24 |
Stage 2 self-distillation improves this coverage of issues, and Stage 3 GRPO retains the discriminative gains of SFT (Tables 3 and 4).
We sincerely thank the reviewer again for the detailed feedback and hope our responses address the concerns. We would be glad to answer any further questions.
References
[4] Saeki et al. "UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022." Interspeech 2022.
[5] Reddy et al. "DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors." ICASSP 2021.
[6] Reddy et al. "DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors." ICASSP 2022.
Author Response to Reviewer uB3K (1/3)
We thank the reviewer for the detailed review, and for recognizing our carefully designed three-stage training pipeline. We address each point below.
W1. Dataset scale and contribution
We agree that ProQA contains fewer original recordings than QualiSpeech. However, we would like to clarify that our dataset's primary contribution is the dense, expert annotations by professional audio engineers, with labels across 18 diverse axes that are much more fine-grained than those of QualiSpeech. We summarize below the key contributions that differentiate ProQA from other audio-quality assessment datasets: