- Rating
- 3
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper introduces SpeechCritic, a framework for training diagnostic speech judges from limited human preference annotations. The task involves reference-conditioned cross-lingual speech comparisons, requiring a model to predict an overall preference, five dimension-level verdicts, and evidence-grounded rationales. The proposed select–calibrate–scale pipeline uses approximately 300 human-annotated comparisons to identify reliable acoustic metrics, calibrate them into probabilistic preference hints, and guide a frontier audio-language model in generating large-scale training supervision. The paper further investigates supervised fine-tuning (SFT), on-policy distillation (OPD), and reinforcement learning (RL) for training a Qwen2.5-Omni-7B judge. Experiments on English–Japanese and English–Spanish comparisons examine judgment accuracy, human agreement, rationale quality, and cross-lingual transfer.
Critical strengths
-
Moving beyond scalar speech-quality scores toward multidimensional comparisons and acoustically grounded explanations is valuable for evaluating speech generation. The reference-conditioned setting and five-dimensional rubric provide a reasonably well-defined evaluation task.
-
The human-calibrated supervision approach is promising. The proposed method uses limited human annotations to select and calibrate acoustic evidence.
-
This paper presents useful analyses of different training strategies. The SFT, OPD, and RL comparisons reveal meaningful trade-offs between Overall accuracy and dimension-level judgments.
Critical weaknesses
-
The contribution of human-calibrated supervision is not sufficiently isolated. Table 2 shows that calibrated hints improve the machine labeler's dimensional judgments. However, Table 3 lacks a matched student trained on direct Gemini supervision without calibrated hints. Therefore, it remains unclear how much the proposed calibration contributes to the final judge's performance. Comparisons with simpler alternatives, such as adjusting TIE priors or providing uncalibrated acoustic hints, would further strengthen the contribution.
-
Better Overall accuracy does not necessarily translate into better diagnostic quality. In Table 3, gated RL improves Overall accuracy from 69.54% to 71.84%, but dimensional macro-F1 decreases from 52.37% to 47.64%. Similarly, privileged OPD with the Qwen3 teacher achieves 73.79% Overall accuracy but only 44.61% dimensional macro-F1.
Questions for authors
-
Can the authors compare students trained on calibrated versus uncalibrated Gemini supervision under identical settings to isolate the contribution of the proposed calibration pipeline?
-
Given that OPD and RL improve Overall accuracy while sometimes reducing dimensional macro-F1, how can the proposed training strategies improve diagnostic quality without sacrificing dimension-level performance?
2 comments
Author Response to XmEE (2/2) Discussion
2. Balancing Overall accuracy and diagnostic quality (Weakness 2, Q2)
We thank the reviewer for highlighting this important trade-off. We agree that higher Overall accuracy alone does not establish better diagnostic quality. We’d like to emphasize that the dimensional decline is not an inevitable cost of OPD or RL; it depends on the training configuration. Our ablations provide three concrete findings that clarify how to obtain a more balanced judge:
1. Choose the OPD teacher for dimensional ability, not just Overall accuracy. With the SFT teacher, OPD increases Overall accuracy from 69.54% to 71.72%, while dimensional macro-F1 remains nearly unchanged (52.37% → 52.15%). In contrast, the Qwen3 teacher increases Overall accuracy by 4.25 points (95% CI [0.57, 7.93]) but reduces dimensional macro-F1 by 7.76 points (95% CI [−9.92, −5.49]). This is consistent with Qwen3’s weaker zero-shot dimensional performance in Table 9. Thus, the teacher yielding the strongest Overall decisions is not necessarily the best teacher for diagnostic judgments.
2. Design RL rewards around the diagnostic outcomes to preserve. Overall-only RL maintains dimensional macro-F1 at 52.79%, whereas gated RL improves Overall accuracy but reduces dimensional performance. Prompted by reviewer 6LJz, we revisited a single-run exploratory Tie-inclusive reward. Compared with gated RL, this run achieves higher dimensional macro-F1 (47.64% → 53.04%), including both A/B and Tie class-F1, and lower Tie-rate MAE (27.30 → 16.29), but lower Overall accuracy (71.84% → 67.93%). This illustrates a configuration-dependent trade-off rather than a universal limitation of RL.
3. If a single balanced recipe is desired, we recommend composed training. Our composed model improves Overall accuracy over SFT by 4.37 points (95% CI [0.57, 8.16]), while retaining SFT-level dimensional performance in the observed means. Its dimensional macro-F1 also significantly exceeds both standalone gated RL and Qwen3-teacher OPD.
| Training strategy | Overall Acc. ↑ | Dim. Macro-F1 ↑ |
|---|---|---|
| SFT | 69.54 | 52.37 |
| Gated RL | 71.84 | 47.64 |
| Privileged OPD (Qwen3 teacher) | 73.79 | 44.61 |
| Composed OPD (SFT teacher) → RL | 73.91 | 52.87 |
Our experiments therefore provide more than evidence of a trade-off: they identify configurations that avoid the large dimensional losses highlighted by the reviewer. Teacher choice, reward design, and training composition are essential to building a balanced diagnostic judge. We have revised Section 5 accordingly.
Author Response to XmEE (1/2) Additional Experiments
We appreciate the reviewer’s suggestion of the direct baseline and their thoughtful, open-ended question about balancing Overall accuracy with diagnostic quality.
1. SFT with Uncalibrated Gemini Supervision (weakness 1 & question 1)
We agree that this is a direct baseline and have now added it. We trained SFT students using uncalibrated Gemini 3.1 Pro supervision without calibrated hints, on exactly the same audio comparisons as the original calibrated supervision. We kept the training configuration, seeds, and DEV-based checkpoint selection identical. Results are mean ± SD across three seeds. We also report 95% paired-bootstrap confidence intervals (20,000 resamples of the same 290 TEST comparisons), averaging seed-wise metrics within each condition before computing differences. Following suggestions from Reviewer F17h, we also added two simple baselines.
Calibration improves the trained student's dimensional macro-F1 by 3.59 points and reduces Tie-rate MAE by 7.74 points, with both confidence intervals excluding zero, with no detectable difference in Overall accuracy. The largest dimensional gains occur in Speaker (+10.94, CI [6.72, 15.02]) and Pronunciation/Accent (+7.46, CI [2.66, 12.08]) dimensions. These results confirm that the benefits of human calibration transfer from the machine labeler to the trained 7B student, improving alignment with human dimensional judgments rather than Overall accuracy. We have added the new baselines to Table 3 and summarized the student-level calibration benefits in the final paragraph of Section 4.3.