acceptodds
Under review as a conference paper at ICLR 2027

AuditEar: Expert Audio Quality Assessment with Large Audio Language Models

Abstract

Existing speech-quality assessment systems typically reduce a recording to a scalar score, offering little insight into what is wrong, how severe the problem is, or how it should be fixed. We introduce ProQA, a dataset of 634 speech recordings assessed by professional audio engineers across 18 production-quality axes. Its 11,412 axis-level annotations pair ordinal severity ratings with issue tags, evidence-grounded assessments, and recommended fixes. To expand this costly expert supervision, we apply controlled augmentations across ten axes and compound their severity increments onto the original expert ratings, producing 15,524 audio variants with known quality changes. We use this corpus to train AuditEar, an audio-language model that diagnoses quality issues and recommends corrective actions. Our three-stage training paradigm combines Supervised Fine-Tuning, On-Policy Self-Distillation from privileged expert context, and GRPO with Verifiable Rewards. On a held-out test set, AuditEar achieves relative improvements of 41.2% in per-axis severity accuracy, 64.1% in pairwise accuracy, and 86.4% in holistic fix-F1 over its zero-shot backbone. It also outperforms Audio-Flamingo-Next, Kimi-Audio-7B-Instruct, and MiDashengLM-7B across all evaluated tasks, with gains of up to 2× over the strongest of these baselines. On external benchmarks, AuditEar achieves 3×–45× the correlation of published zero-shot audio-language baselines on QualiSpeech and reaches 74.3% preference accuracy on SpeechEval, compared with 70.4% for audio-language models evaluated zero-shot. These results show that expert, per-axis supervision can turn audio-quality assessment into a grounded and actionable diagnosis.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
audio quality assessment, speech quality assessment, audio language models, controlled audio augmentation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uB3KOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AuditEar, a speech quality evaluation LLM fine-tuned from Qwen3-Omni-30B-A3B-Instruct using the proposed ProQA dataset. ProQA consists of 634 speech recordings annotated across 18 dimensions of speech quality. The authors introduce a multi-stage training pipeline involving SFT, RGSD+OPSD, and GRPO. Experimental results demonstrate promising in-domain performance and cross-domain generalization on the QualiSpeech and SpeechEval benchmarks.

Critical strengths

The training pipeline is carefully designed, incorporating multiple training stages to further exploit the potential of speech LLMs for speech quality evaluation. The combination of different training strategies provides a systematic approach to improving the model's evaluation capabilities and generalization.

Critical weaknesses
  1. Limited dataset scale and contribution. Although the authors acknowledge the relatively small scale of ProQA and emphasize the difficulty of collecting human annotations, the dataset size remains a significant concern. For comparison, the existing QualiSpeech dataset contains more than 10,000 speech samples and over 70,000 human annotations, substantially exceeding the scale of ProQA. Given that considerably larger speech quality evaluation datasets already exist, the incremental contribution of ProQA appears somewhat limited.

  2. Limited cross-domain generalization and practical applicability. Despite the improvements achieved through the proposed multi-stage training pipeline, AuditEar still underperforms models fine-tuned on the target domains when evaluated on out-of-domain datasets such as QualiSpeech and SpeechEval. This suggests that the limited scale of the training data may constrain the model's generalization capabilities. In practical scenarios involving diverse and previously unseen speech samples, AuditEar may therefore struggle to outperform models trained on larger speech quality evaluation datasets. This limitation raises concerns about the practical applicability of AuditEar as a general-purpose speech quality evaluator.

Questions for authors
  1. Could the authors provide additional comparisons with SSL-based speech quality assessment methods, such as UTMOS [1], particularly on rating-based evaluation tasks? Such comparisons would help establish the effectiveness of AuditEar relative to established speech quality prediction approaches.

  2. According to Table 3, the SFT model always underperforms the base Qwen3-Omni model. Could the authors provide further insights into this observation? In particular, why does supervised fine-tuning lead to a performance degradation despite using task-specific speech quality evaluation data?

[1] Saeki, Takaaki, et al. "Utmos: Utokyo-sarulab system for voicemos challenge 2022." arXiv preprint arXiv:2204.02152 (2022).

6 comments
AuthorsOpenReview ↗

Author Response to Reviewer uB3K (1/3)

We thank the reviewer for the detailed review, and for recognizing our carefully designed three-stage training pipeline. We address each point below.

W1. Dataset scale and contribution

We agree that ProQA contains fewer original recordings than QualiSpeech. However, we would like to clarify that our dataset's primary contribution is the dense, expert annotations by professional audio engineers, with labels across 18 diverse axes that are much more fine-grained than those of QualiSpeech. We summarize below the key contributions that differentiate ProQA from other audio-quality assessment datasets:

  • Amount of supervision. ProQA's expert annotations yield 79,925 training question–answer pairs across nine tasks before augmentation (Stage 1). Controlled augmentation grounded on these annotations adds 15,524 audio variants, bringing the audio count to a comparable scale. This gives 98,469 training examples across the three stages (Table 2). QualiSpeech's annotations, by contrast, consist mainly of seven 1–5 scores per clip, plus short notes on noise, distortion, and pauses.
  • ProQA covers a wider breadth of audio quality axes. ProQA rates 18 production-quality axes per recording, such as mic quality and placement, handling noise and pops, reverberation, flutter echoes, stationary and dynamic noise, sibilance, distortion, loudness and processing artefacts (Table 13, Appendix A.6). Many of these are rarely covered by existing datasets, and they give finer-grained coverage than QualiSpeech's seven aspects (noise, distortion, speed, continuity, listening effort, naturalness and overall quality).
  • ProQA has a greater depth of expert annotations. Every ProQA axis contains expert-annotated issue tags, an ordinal severity, a written assessment grounded in audible evidence, and recommended fixes. In total, the engineers wrote 11,412 severity ratings, 11,412 issue-tag annotations, 10,141 assessments and 8,823 fixes. In QualiSpeech, the natural-language descriptions are generated by GPT-4o-mini from the listeners' scores and notes and then revised by annotators, and QualiSpeech provides no recommended fixes. This rich supervision in ProQA supports capabilities beyond quality scoring and provides a foundation for broader audio assessment and editing workflows, such as holistic assessment, targeted issue diagnosis and actionable remediation guidance.
  • Annotator expertise. ProQA was annotated by 22 professional audio engineers with at least five years of mixing and mastering experience, each spending about half an hour per recording. QualiSpeech was annotated by 25 listeners at about six minutes per clip.
  • Real recordings. All ProQA recordings are real human speech from different datasets and recording conditions, covering a range of microphones, room acoustics and processing characteristics. In comparison, 49% of QualiSpeech audio is synthetic speech from TTS and voice-conversion systems.
AuthorsOpenReview ↗

Author Response to Reviewer uB3K (2/3)

W2. Cross-domain generalization and practical applicability

We would like to highlight that the models the reviewer refers to are fine-tuned on the QualiSpeech and SpeechEval datasets themselves.

  • In-domain vs. out-of-domain. By out-of-domain generalization, we mean that AuditEar is never trained on QualiSpeech or SpeechEval data. It is trained only on the ProQA training split, which differs substantially from the training splits of both benchmarks, as described in our response to W1. ProQA covers different quality axes (such as mic quality and placement, flutter echoes and sibilance), different tasks (per-axis severity, issue tags, written assessments and recommended fixes), and different question formats. QualiSpeech and SpeechEval instead use different axes and task formats from ProQA. In contrast, QualiSpeech-SALMONN is fine-tuned on QualiSpeech's training data and SQ-LLM is fine-tuned on SpeechEval's data, so their scores on their own benchmarks are in-domain results.
  • Comparison with zero-shot models. The fairer comparison is with models that have also not been trained on these out-of-domain benchmarks. AuditEar performs much better on the QualiSpeech benchmark than other zero-shot audio-language models evaluated in the QualiSpeech paper, such as Qwen-Audio, Qwen2-Audio and SALMONN, whose best results we list in Table 5. AuditEar also improves on its base model, Qwen3-Omni, from 0.169 to 0.628 on QualiSpeech and from 0.564 to 0.743 on SpeechEval, as shown in the table below. Prior work also treats generalization to unseen domains as a separate and harder setting, in which quality predictors typically fall below models trained on the target data [3]. Since our goal was to test generalization to out-of-domain datasets, we did not add any of QualiSpeech's or SpeechEval's data to AuditEar's training.

We further compared AuditEar with these two audio-quality assessment judges, QualiSpeech-SALMONN and SQ-LLM, on all three benchmarks, each evaluated on the same clips with its own prompt format (Appendix A.10).

ModelTrained onQualiSpeech (Overall Quality)SpeechEval (Pairwise Comparison)ProQA (Pairwise Comparison Score)
Qwen3-Omni–0.1690.5640.574
QualiSpeech SALMONN-7B [1]QualiSpeech0.7120.6960.689
SQ-LLM [2]SpeechEval0.5890.7930.713‾\underline{0.713}
AuditEarProQA0.628‾\underline{0.628}0.743‾\underline{0.743}0.886
  • Trained only on ProQA, AuditEar ranks second on both out-of-domain benchmarks, behind only the model fine-tuned on that benchmark's own data. AuditEar scores 0.628 on QualiSpeech and 0.743 on SpeechEval. AuditEar also increases the scores of its base model, Qwen3-Omni, from 0.169 to 0.628 on QualiSpeech and from 0.564 to 0.743 on SpeechEval, which shows the value of ProQA's expert annotations.
  • Although QualiSpeech-SALMONN and SQ-LLM are trained on more audio and annotations than AuditEar, each of them comes third on the other external benchmark. QualiSpeech-SALMONN scores 0.696 on SpeechEval, against 0.743 for AuditEar, and SQ-LLM scores 0.589 on QualiSpeech, against 0.628 for AuditEar. On ProQA they reach only 0.689 and 0.713, against 0.886 for AuditEar. Since Qwen3-Omni without ProQA training scores lowest in every column, the backbone alone does not explain this, which highlights the importance of the high-quality expert annotations in ProQA.
  • On SpeechEval, AuditEar matches SQ-LLM on English, and the remaining gap comes from other languages. SpeechEval covers English, French, Japanese and Chinese, and SQ-LLM is trained on all four languages, whereas ProQA focuses on English speech. On the same SpeechEval test pairs split by language, the results are:
ModelEnglishFrenchJapaneseChinese
Qwen3-Omni0.5870.6030.5250.541
SQ-LLM0.729‾\underline{0.729}0.8260.7780.832
AuditEar0.7360.799‾\underline{0.799}0.730‾\underline{0.730}0.719‾\underline{0.719}

On English, AuditEar slightly outperforms SQ-LLM without any SpeechEval training (0.736 vs. 0.729). The gap is in French, Japanese and Chinese, yet even there AuditEar improves on its base model by 0.18–0.21. We have added the cross-benchmark and per-language results to the paper (Appendix A.10).

References

[1] Wang et al. "QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptions." ACL 2025.

[2] Wang et al. "SpeechLLM-as-Judges: Towards General and Interpretable Speech Quality Evaluation." ACL 2026.

[3] Cooper et al. "Generalization Ability of MOS Prediction Networks." ICASSP 2022.

AuthorsOpenReview ↗

Author Response to Reviewer uB3K (3/3)

Q1. Comparison with UTMOS and other quality predictors

We have evaluated UTMOS [4] and DNSMOS [5, 6] on ProQA.

  • UTMOS [4] predicts a mean opinion score from self-supervised speech features, and DNSMOS predicts MOS-style scores for overall speech quality (P.808 [5]) and for signal, background and overall quality (P.835 [6]). Both output numerical scores for each recording.
  • Due to these models' limitations, such as producing only numerical scores and no per-axis severity ratings, issue tags, written assessments or recommended fixes, they cannot be fully evaluated on all the tasks of our benchmark.

We therefore evaluated them on a subset of ProQA on pairwise comparison, taking the recording with the higher predicted score as the better one. AuditEar scores higher than UTMOS and DNSMOS. In addition to choosing the better recording in each pair, AuditEar also explains why it is better, and identifies the issues and recommends fixes for all axes, which these predictors cannot do. We have added this comparison to the paper (Appendix A.11).

ModelPairwise Comparison Score
AuditEar0.886
UTMOS [4]0.718
DNSMOS P.808 [5]0.713
DNSMOS P.835 overall [6]0.755‾\underline{0.755}

Q2. Stage-1 SFT vs. the base model

Across Tables 3 and 4, Stage-1 SFT actually outperforms the base Qwen3-Omni model on the discriminative tasks (severity accuracy, issue detection, pairwise accuracy and position consistency), but it underperforms it on the free-text metrics (issue-, tag- and fix-F1). This can be explained as follows:

  • To evaluate the free-text metrics, an LLM judge first extracts keywords from the model response and the ground truth answer and then matches them. The base model writes long, generic responses, as shown by the fact that, on average, its responses list 3.6 issues against 2.5 in the ground truth. Many of its items do not match, so its precision is low, but because it lists so many keywords, enough items match to give it high recall.
  • Stage-1 SFT writes short, specific answers that are usually correct, so its precision is high. However, it under-reports some errors, so its recall is low. We have added this precision/recall analysis to the paper (Appendix A.12).

The precision and recall scores of the base Qwen3-Omni and Stage-1 SFT are shown below:

MetricStage-1 SFT PrecisionStage-1 SFT RecallQwen3-Omni PrecisionQwen3-Omni Recall
Single-Axis · issue0.590.210.410.60
Single-Axis · tag0.600.200.330.52
Single-Axis · fix0.450.130.340.54
Holistic · issue0.600.190.540.24

Stage 2 self-distillation improves this coverage of issues, and Stage 3 GRPO retains the discriminative gains of SFT (Tables 3 and 4).

We sincerely thank the reviewer again for the detailed feedback and hope our responses address the concerns. We would be glad to answer any further questions.

References

[4] Saeki et al. "UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022." Interspeech 2022.

[5] Reddy et al. "DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors." ICASSP 2021.

[6] Reddy et al. "DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors." ICASSP 2022.

Reviewer_yEQQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents ProQA dataset together with the AuditEar, which is a comprehensive model to judge the audio quality. The training consists of 3 stages, which leverage small-scale, professionally labeled data. The experiments shows encouraging results in in-domain and out-domain setups.

Critical strengths

(1) The paper presents the ProQA, which is professionally labeled. As long as that resources can be open-sourced to the public, it would be a great contribution to the community. (2) The experiments also progressively show the performance improvement across the 3 stages, each of which shows clear improvement. (3) The model also transfers well to the unseen benchmarks.

Critical weaknesses

(1) Although the results in table 3 and 4 are encouraging, they are based on the zero-shot naive baselines. This is somehow understandable because there are rare models developed on this purpose. (2) The data augmentation in Section 3.2 is generally simple and mostly based on signal processing. Detections in those directions can be easily achieved even without those ProQA data, and can possibly be more scalable than the current setups. (3) A series of works in evaluation might be missing: Uni-VERSA: Versatile Speech Assessment with a Unified Network ARECHO: Autoregressive evaluation via chain-based hypothesis optimization for speech multi-metric estimation

Questions for authors

(1) I wonder if the carefully curated ProQA will go public - for this work, the open-source will be the pivot of its value (2) Is there a more detailed description about the labeler background and how they are trained to produce the ProQA dataset.

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer yEQQ (1/2)

We thank the reviewer for their constructive review and for noting that AuditEar transfers well to unseen benchmarks.

W1. Zero-shot baselines

We agree that Tables 3 and 4 primarily involve zero-shot baselines, as few models are designed for this task. However, these models are also standard reference points in audio-quality assessment:

  • QualiSpeech [1] and SpeechEval [2] also evaluate models such as Qwen-Audio, Qwen2-Audio and Qwen2.5-Omni in zero-shot settings.
  • Audio-Flamingo-Next [3] and Kimi-Audio [4] are designed as audio foundation models with advanced audio perception, understanding and reasoning capabilities. Evaluating them examines whether these capabilities extend to audio-quality assessment without task-specific training.
  • Tables 3 and 4 (Section 5.1) show that these models still struggle to assess multiple recording-quality axes jointly. This highlights the gap between general-purpose audio understanding and expert multi-axis quality assessment, and the importance of ProQA's data and AuditEar's training design.

Beyond zero-shot models, we compare AuditEar with two more types of models:

1. Dedicated speech-quality judges. We compare AuditEar with the audio-LLM judges SQ-LLM [2] and QualiSpeech's SALMONN-7B [1] on ProQA pairwise comparison:

ModelProQA (Pairwise Comparison Score)
QualiSpeech SALMONN-7B [1]0.689
SQ-LLM [2]0.713‾\underline{0.713}
AuditEar0.886

Existing audio-LLM judges lack the expertise to match professional production-quality judgments, which highlights the gap that ProQA fills. We have added this comparison to the paper (Appendix A.10).

2. SFT on our data (ablation already in the paper). Starting from the same Stage-1 checkpoint, we train Stage 2 with either SFT or our self-distillation (RGSD + OPSD) (Section 5.3, Appendix Table 8). Compared with continued SFT, self-distillation substantially improves performance on the open-ended assessment and fix tasks.

W2. Signal-processing augmentation

We agree that the augmentation operations are signal-processing-based, and that individual acoustic issues can sometimes be simulated and learned with specialized methods without ProQA. However, our augmentation builds on ProQA's expert annotations, which are essential as explained below:

  • Expert judgments provide the grounding and calibration reference. We augment recordings that professional audio engineers have already annotated, and update the severity of each augmented recording relative to its original expert severity (Section 3.2, Appendix A.7). Unlike prior work such as ParA-LLM [5], where acoustic labels come entirely from simulation, our approach is more grounded, as it starts from professional perceptual judgment rather than from heuristics alone. ProQA also retains the engineers' assessments of issues already present in the recordings, so the model learns from real audio defects and not only from simulated ones.
  • ProQA enables assessment beyond single-issue detection. Specialized methods may detect individual acoustic issues without ProQA, but AuditEar assesses all 18 quality axes jointly and explains the issues in each, with severity ratings and recommended fixes. General audio-language models such as Audio-Flamingo-Next and Kimi-Audio, trained on large-scale data without this expertise, still struggle with this (Tables 3 and 4, Section 5.1).
  • Axes with augmentation score higher than axes without it. The ten augmented axes outperform the eight axes trained only on the original recordings, in both issue identification (issue-F1 0.546 vs. 0.500) and recommended fixes (fix-F1 0.507 vs. 0.428) (Section 5.2). These results show that even simple signal-processing augmentations work well when they build upon expert annotations.
  • Scalability. We agree that this approach can be scaled further. In this paper we cover 10 of the 18 axes with augmentation, and additional transformations can extend the coverage in the same way, each building on the original professional annotations as the calibration reference.

References

[1] Wang et al. "QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptions." ACL 2025.

[2] Wang et al. "SpeechLLM-as-Judges: Towards General and Interpretable Speech Quality Evaluation." ACL 2026.

[3] Ghosh et al. "Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music." arXiv:2604.10905, 2026.

[4] KimiTeam et al. "Kimi-Audio Technical Report." arXiv:2504.18425, 2025.

[5] Anand et al. "ParA-LLM: A Unified Approach to Paralinguistic and Acoustic Speech Understanding." Interspeech 2026.

AuthorsOpenReview ↗

Author Response to Reviewer yEQQ (2/2)

W3. Uni-VERSA and ARECHO

We have added both to the Related Work (Section 2) and evaluated both on ProQA.

  • Uni-VERSA [6] predicts a range of speech-assessment metrics, such as naturalness, intelligibility and noise, and ARECHO [7] extends this to 87 metrics, estimated autoregressively. Both output numerical scores for predefined metrics.
  • Due to these models' limitations, such as producing only numerical scores and no per-axis severity ratings, issue tags, written assessments or recommended fixes, they cannot be fully evaluated on all the tasks of our benchmark.

We therefore evaluated them on a subset of ProQA on pairwise comparison, using their public checkpoints and taking the recording with the higher predicted score as the better one.

ModelPairwise Comparison Score
AuditEar0.886
Uni-VERSA [6]
    MOS0.750
    UTMOS0.782
    SHEET SSQA0.787
    DNSMOS P.8080.798
ARECHO [7] (Base / Scaled)
    NISQA-MOS0.851‾\underline{0.851} / 0.824
    SCOREQ0.811 / 0.832
    SHEET SSQA0.809 / 0.827
    DNSMOS P.835 overall0.803 / 0.809
    DNSMOS P.8080.729 / 0.758
    UTMOS0.745 / 0.713
    UTMOSv20.779 / 0.771

AuditEar scores higher than Uni-VERSA and both variants of ARECHO. In addition to choosing the better recording in each pair, AuditEar also explains why it is better.

We have added this comparison to the paper (Appendix A.11).

Q1. Public release

We thank the reviewer for raising this. We are currently working through the internal approval process. We will make our best effort to release the dataset.

Q2. Annotator background and training

We request the reviewer to kindly refer to Appendix A.4, which describes the annotators' background and training in detail. We summarize the key points below:

  • Background. ProQA was annotated by 22 professional audio engineers, each with at least five years of experience producing, mixing or mastering dialogue-centric audio. We recruited them based on their portfolios, an intake questionnaire and their experience with professional tools such as Pro Tools, DaVinci Resolve or Adobe Audition.
  • Training and quality control. Each engineer completed a calibration pass of about four hours, with written guidelines and eight low and high-quality anchor recordings. To keep annotation quality and rubric use consistent, we then used a three-stage quality-control process.
    1. Pre-screening. Each engineer's first batch served as a quality test, which the authors reviewed in detail. Only engineers whose assessments were detailed and specific went on to annotate more data.
    2. Mid-screening and active QC. During annotation, we continuously reviewed responses, gave feedback, and calibrated the use of the rubric when necessary. We notified annotators whose responses were brief, generic or did not meet our standards.
    3. Post-screening. After collection, we removed annotations whose explanations were too brief or lacked specific, actionable detail, about 10.2% of the total.

We sincerely thank the reviewer again for their feedback.

References

[6] Shi et al. "Uni-VERSA: Versatile Speech Assessment with a Unified Network." Interspeech 2025.

[7] Shi et al. "ARECHO: Autoregressive Evaluation via Chain-Based Hypothesis Optimization for Speech Multi-Metric Estimation." NeurIPS 2025.

Reviewer_ETEfOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a speech audio quality benchmark PROQA and an ALM judge AUDITEAR that is trained to evaluate speech audio quality. A collection of 634 English-language speech audios are taken from a variety of existing open-source and closed-source datasets, covering both real and simulated audio and covering a variety of scenarios and settings. These are each annotated along 18 axes by professional audio engineers. Axes include audio/speech quality issues like microphone quality, room reverb, noise types, tonal balance, equalization, etc. Each axis is annotated with rating, a description, and suggested fixes. The dataset is augmented by synthetically increasing the severity of the existing issues and using an LLM to update the text-based annotation fields. Nine different training tasks are formulated from this expanded dataset, and SFT + On-Policy Self-Distillation + RLVR is used to train a Qwen3 ALM judge. This trained judge is able to outperform the original Qwen3 judge as well as several other off-the-shelf ALM judges on PROQA, as well as on QualiSpeech and SpeechEval which are out-of-domain. It is also able to perform competitively on SpeechEval against a model that was specifically finetuned on a training split of SpeechEval.

Critical strengths
  1. PROQA is annotated in a very detailed manner using 22 expert annotators (professional audio engineers), which I think is a commendable effort.

  2. AUDITEAR is able to outperform the baselines on PROQA, and the performance generalizes to other QualiSpeech and SpeechEval.

  3. The paper is well-written and easy to follow.

Critical weaknesses
  1. As mentioned in the limitations section, each example is only seen by one annotator, which makes it impossible to measure inter-annotator agreement.

  2. Only English-language speech is included. I don’t think this is a showstopper, but worth noting.

Questions for authors

I don’t have any questions at the moment that I think would affect my assessment. But I’ll take a look at the other reviewers’ feedback in case there is anything I might have overlooked.

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer ETEf

We thank the reviewer for their positive assessment and for recognizing the value of our ProQA dataset and the 3-stage training paradigm of AuditEar.

W1. One annotator per recording

We agree that one engineer per recording prevents us from measuring inter-annotator agreement, as acknowledged in Section 6. To keep annotation quality and rubric use consistent, we used a three-stage quality-control process:

  1. Pre-screening. We recruited professional audio engineers based on their portfolios and experience (Appendix A.4), and used each engineer's first batch of annotations as a quality test.
  2. Mid-screening and active QC. Each engineer completed a calibration pass with written guidelines and eight low and high-quality anchor recordings (Appendix A.4). During annotation, we continuously monitored responses, gave feedback on rubric use, and notified annotators whose responses did not meet our standard.
  3. Post-screening. We removed annotations whose assessments were too brief or unspecific, about 10.2% of the total.

As a consistency check, we grouped the recordings into four quality tiers by the known quality of their source datasets. Each tier's mean overall rating preserves the expected ranking:

SourceMean overall rating (1–5)
High-quality studio recordings4.21
Public read and expressive corpus3.51
Crowd-sourced and in-the-wild audios2.69
Low-quality recordings1.45

Each engineer's axis-level severities also agree with the overall rating they assigned independently: the within-annotator rank correlation has a median of 0.92 and ranges from 0.81 to 0.95 (Section 3.1). We have added these extra details in Appendix A.4.

W2. English-only speech

ProQA primarily focuses on English, and we agree that broader language coverage is desirable. Although annotators were instructed to judge recording and production quality rather than the spoken content, familiarity with a language can still influence some axes, particularly intelligibility. We therefore focused on English, to match our predominantly English-speaking annotator pool and to avoid mistaking unfamiliarity with a language for audio-quality degradation. Extending ProQA to other languages with audio engineers who are native speakers of those languages is an important direction for future work (Section 6).

We thank the reviewer again for their encouraging review.

Thank you for your rebuttal. I have looked at it, as well as the other reviews and the discussion. I still think my positive overall rating is appropriate so I will maintain it.

We thank the reviewer for taking the time to go through our rebuttal and the discussion, and for maintaining their positive assessment of our work. We greatly appreciate their feedback throughout the review process.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

We thank the ACs and all reviewers for their thoughtful feedback on our paper. Their comments helped us strengthen our paper.

Reviewers recognized several strengths of our work, which we summarize below:

  1. Expert-annotated dataset. Reviewers found ProQA's detailed annotation by 22 professional audio engineers a commendable effort (ETEf), and a great contribution to the community (yEQQ).
  2. Systematic multi-stage training pipeline. Reviewers found our three-stage training pipeline carefully designed and systematic (uB3K), with a clear improvement at each stage (yEQQ).
  3. Generalization to unseen benchmarks. Reviewers noted that AuditEar outperforms other state-of-the-art models on ProQA and also performs well on QualiSpeech and SpeechEval, which are out-of-domain benchmarks (ETEf, yEQQ).
  4. Clear presentation. Reviewers found the paper well written and easy to follow (ETEf), and rated its presentation clear (ETEf, yEQQ).

During the rebuttal, reviewers raised some points. We ran new experiments to provide conclusive evidence, and pointed the reviewers to the relevant sections of the paper to clarify their concerns. We have added these to the paper, marked in blue. Here are the key highlights from the discussion:

  1. Comparison with other audio quality judges (yEQQ, uB3K). We compared AuditEar with two Audio-LLM quality judges, QualiSpeech-SALMONN and SQ-LLM, and with scalar quality predictors: UTMOS, DNSMOS, Uni-VERSA and ARECHO. On ProQA, AuditEar outperforms the two Audio-LLM judges by 28.6% and 24.3% respectively, and the scalar predictors by 4.1% to 24.3%.
  2. Cross-domain generalization (uB3K, yEQQ). AuditEar is trained only on ProQA, so QualiSpeech and SpeechEval are out-of-domain benchmarks for it. On each of these benchmarks, AuditEar ranks second-best among all Audio-LLMs, behind only the model trained on that benchmark itself. AuditEar outperforms the other models by at least 6.6% on QualiSpeech and 6.8% on SpeechEval. Following uB3K's suggestion, we also compared it with the scalar predictors. On QualiSpeech, AuditEar performs 19.6% better than UTMOS and 56.6% to 97.5% better than DNSMOS, and on SpeechEval it performs competitively with them.
  3. Annotator details (ETEf, yEQQ). We provided the reviewers with additional details about the annotators and our quality-control process, and added them to Appendix A.4, which describes annotator recruitment, our active quality control and each stage of screening and dataset creation in detail.
  4. Dataset scale and contribution (uB3K). We clarified that ProQA's main contribution is the density of expert supervision. For each of the 18 axes, the audio engineers annotated multiple details, such as a severity rating, issue tags, a written assessment and recommended fixes, which lead to 79,925 training question-answer pairs. To our knowledge, no existing speech-quality dataset provides annotations of this depth. ProQA also covers axes that existing datasets rarely include, such as mic quality and placement, handling noise and pops, flutter echoes and sibilance.

We again appreciate the effort that the reviewers put into reviewing our work. We hope that this summary helps the Area Chairs in making their decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.