acceptodds
Under review as a conference paper at ICLR 2027

SpeechCritic: Learning a Diagnostic Speech Judge from Limited Human Preferences

Abstract

Human speech conveys rich perceptual information, such as emotion and speaker identity, yet most automatic speech quality judges reduce it to a single naturalness score. We study diagnostic speech judges: given two candidates, a diagnostic judge decides which is better, along which perceptual dimensions (e.g., timbre, emotion, timing) they differ, and which audible cues support its decision. Learning such judges is challenging: expert annotation is costly, and simply prompting a frontier audio-language model to produce labels is unreliable: our probing reveals substantial errors and unstable instruction following. We introduce SpeechCritic, which learns a diagnostic judge in a reference-conditioned cross-lingual setting from only about 300 human-labeled comparisons. Rather than replacing the frontier model, SpeechCritic calibrates it with these labels: for each dimension, it selects the acoustic measurements that agree with human judgments, maps them to A/Tie/B probabilities, and passes these to the model as non-binding hints alongside the audio. Compared with the same model labeling without hints, this raises dimension-level agreement with humans by 6.3 points and cuts the mismatch with human Tie rates by 10.4 points. We then train a 7B judge on this supervision and find that different training signals shape different judge behaviors: supervised fine-tuning establishes the task, on-policy distillation transfers the teacher's dimension-level strengths and weaknesses, and reinforcement learning helps most on clear-cut comparisons where human raters agree. Notably, human listeners also find that reinforcement learning makes rationales cite more specific, localized acoustic cues, although it never directly rewards rationale text. Finally, we show that the pipeline is language-pair agnostic by instantiating it on both English–Japanese and English–Spanish. Together, these results demonstrate a path from limited human preferences to a diagnostic speech judge.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
speech evaluation, llm-as-a-judge, multimodal evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XmEEOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper introduces SpeechCritic, a framework for training diagnostic speech judges from limited human preference annotations. The task involves reference-conditioned cross-lingual speech comparisons, requiring a model to predict an overall preference, five dimension-level verdicts, and evidence-grounded rationales. The proposed select–calibrate–scale pipeline uses approximately 300 human-annotated comparisons to identify reliable acoustic metrics, calibrate them into probabilistic preference hints, and guide a frontier audio-language model in generating large-scale training supervision. The paper further investigates supervised fine-tuning (SFT), on-policy distillation (OPD), and reinforcement learning (RL) for training a Qwen2.5-Omni-7B judge. Experiments on English–Japanese and English–Spanish comparisons examine judgment accuracy, human agreement, rationale quality, and cross-lingual transfer.

Critical strengths
  • Moving beyond scalar speech-quality scores toward multidimensional comparisons and acoustically grounded explanations is valuable for evaluating speech generation. The reference-conditioned setting and five-dimensional rubric provide a reasonably well-defined evaluation task.

  • The human-calibrated supervision approach is promising. The proposed method uses limited human annotations to select and calibrate acoustic evidence.

  • This paper presents useful analyses of different training strategies. The SFT, OPD, and RL comparisons reveal meaningful trade-offs between Overall accuracy and dimension-level judgments.

Critical weaknesses
  • The contribution of human-calibrated supervision is not sufficiently isolated. Table 2 shows that calibrated hints improve the machine labeler's dimensional judgments. However, Table 3 lacks a matched student trained on direct Gemini supervision without calibrated hints. Therefore, it remains unclear how much the proposed calibration contributes to the final judge's performance. Comparisons with simpler alternatives, such as adjusting TIE priors or providing uncalibrated acoustic hints, would further strengthen the contribution.

  • Better Overall accuracy does not necessarily translate into better diagnostic quality. In Table 3, gated RL improves Overall accuracy from 69.54% to 71.84%, but dimensional macro-F1 decreases from 52.37% to 47.64%. Similarly, privileged OPD with the Qwen3 teacher achieves 73.79% Overall accuracy but only 44.61% dimensional macro-F1.

Questions for authors
  • Can the authors compare students trained on calibrated versus uncalibrated Gemini supervision under identical settings to isolate the contribution of the proposed calibration pipeline?

  • Given that OPD and RL improve Overall accuracy while sometimes reducing dimensional macro-F1, how can the proposed training strategies improve diagnostic quality without sacrificing dimension-level performance?

2 comments
AuthorsOpenReview ↗

Author Response to XmEE (1/2) Additional Experiments

We appreciate the reviewer’s suggestion of the direct baseline and their thoughtful, open-ended question about balancing Overall accuracy with diagnostic quality.

1. SFT with Uncalibrated Gemini Supervision (weakness 1 & question 1)

We agree that this is a direct baseline and have now added it. We trained SFT students using uncalibrated Gemini 3.1 Pro supervision without calibrated hints, on exactly the same audio comparisons as the original calibrated supervision. We kept the training configuration, seeds, and DEV-based checkpoint selection identical. Results are mean ± SD across three seeds. We also report 95% paired-bootstrap confidence intervals (20,000 resamples of the same 290 TEST comparisons), averaging seed-wise metrics within each condition before computing differences. Following suggestions from Reviewer F17h, we also added two simple baselines.

MethodOverall Acc. ↑Dim. Macro-F1 ↑Tie-rate MAE ↓
Acoustic-metrics-only model61.7238.7626.01
Human-DEV-only SFT62.07 ± 3.8948.86 ± 10.4814.00 ± 4.83
Uncalibrated Gemini SFT69.66 ± 1.1948.78 ± 0.9026.70 ± 1.71
Hint-calibrated Gemin SFT69.54 ± 0.8052.37 ± 0.1018.97 ± 0.44
Calibrated SFT − Uncalibrated SFT [95% CI]-0.11 [-4.25, 4.02]+3.59 [1.18, 5.91]-7.74 [-9.54, -5.72]

Calibration improves the trained student's dimensional macro-F1 by 3.59 points and reduces Tie-rate MAE by 7.74 points, with both confidence intervals excluding zero, with no detectable difference in Overall accuracy. The largest dimensional gains occur in Speaker (+10.94, CI [6.72, 15.02]) and Pronunciation/Accent (+7.46, CI [2.66, 12.08]) dimensions. These results confirm that the benefits of human calibration transfer from the machine labeler to the trained 7B student, improving alignment with human dimensional judgments rather than Overall accuracy. We have added the new baselines to Table 3 and summarized the student-level calibration benefits in the final paragraph of Section 4.3.

AuthorsOpenReview ↗

Author Response to XmEE (2/2) Discussion

2. Balancing Overall accuracy and diagnostic quality (Weakness 2, Q2)

We thank the reviewer for highlighting this important trade-off. We agree that higher Overall accuracy alone does not establish better diagnostic quality. We’d like to emphasize that the dimensional decline is not an inevitable cost of OPD or RL; it depends on the training configuration. Our ablations provide three concrete findings that clarify how to obtain a more balanced judge:

1. Choose the OPD teacher for dimensional ability, not just Overall accuracy. With the SFT teacher, OPD increases Overall accuracy from 69.54% to 71.72%, while dimensional macro-F1 remains nearly unchanged (52.37% → 52.15%). In contrast, the Qwen3 teacher increases Overall accuracy by 4.25 points (95% CI [0.57, 7.93]) but reduces dimensional macro-F1 by 7.76 points (95% CI [−9.92, −5.49]). This is consistent with Qwen3’s weaker zero-shot dimensional performance in Table 9. Thus, the teacher yielding the strongest Overall decisions is not necessarily the best teacher for diagnostic judgments.

2. Design RL rewards around the diagnostic outcomes to preserve. Overall-only RL maintains dimensional macro-F1 at 52.79%, whereas gated RL improves Overall accuracy but reduces dimensional performance. Prompted by reviewer 6LJz, we revisited a single-run exploratory Tie-inclusive reward. Compared with gated RL, this run achieves higher dimensional macro-F1 (47.64% → 53.04%), including both A/B and Tie class-F1, and lower Tie-rate MAE (27.30 → 16.29), but lower Overall accuracy (71.84% → 67.93%). This illustrates a configuration-dependent trade-off rather than a universal limitation of RL.

3. If a single balanced recipe is desired, we recommend composed training. Our composed model improves Overall accuracy over SFT by 4.37 points (95% CI [0.57, 8.16]), while retaining SFT-level dimensional performance in the observed means. Its dimensional macro-F1 also significantly exceeds both standalone gated RL and Qwen3-teacher OPD.

Training strategyOverall Acc. ↑Dim. Macro-F1 ↑
SFT69.5452.37
Gated RL71.8447.64
Privileged OPD (Qwen3 teacher)73.7944.61
Composed OPD (SFT teacher) → RL73.9152.87

Our experiments therefore provide more than evidence of a trade-off: they identify configurations that avoid the large dimensional losses highlighted by the reviewer. Teacher choice, reward design, and training composition are essential to building a balanced diagnostic judge. We have revised Section 5 accordingly.

Reviewer_6LJzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to build a multi-dimensional, diagnostic speech judge for reference-conditioned cross-lingual speech comparison (comparing two target-language candidates against a source-language reference across Speaker, Emotion, Timing, Pronunciation/Accent, and Audio Artifacts, plus a binary Overall verdict and text rationales) using only ~300 human-labeled examples. Because zero-shot frontier audio-language models severely under-predict ties and hallucinate acoustic explanations, the authors propose a select-calibrate-scale pipeline: they use the ~300 human dev labels to screen automatic acoustic metrics per dimension, fit a multinomial logistic regression to map metric differences into A/TIE/B probabilities, and pass these probabilities as non-binding hints to Gemini 3.1 Pro to label ~10,000 unlabeled pairs. Using this synthetic supervision, they train a 7B student judge (Qwen2.5-Omni-7B) and systematically compare SFT, on-policy distillation (OPD), and RL (DAPO/GRPO), analyzing how each training signal affects overall accuracy, dimensional F1, rationale grounding, and cross-lingual transfer between English-Japanese and English-Spanish.

Critical strengths
  • The paper is refreshingly honest about negative results and trade-offs. Rather than claiming a single method beats everything across the board, the authors transparently report where standard acoustic metrics fail screening (e.g., UTMOS/DNSMOS/NISQA on Audio Artifacts in Table 8), where RL hurts dimensional macro-F1 (Table 3), and where fluent, verbose rationales from OPD mislead human evaluators even when the verdict is wrong (Section 6.2).
  • The select-calibrate-scale idea in Section 4 is a clean and lightweight way to inject domain priors into a frontier audio LLM judge. Screening candidate metrics via grouped out-of-fold validation against human labels and converting them into soft A/TIE/B probabilities noticeably improves Gemini 3.1 Pro's dimensional macro-F1 (+6.3 points) and TIE-rate calibration (-10.4 points MAE) on the test set without overriding the model's ability to listen to the raw audio.
  • The empirical analysis in Sections 5 and 6 covers a lot of useful ground, including unfreezing the audio encoder vs. LLM LoRA (Table 4), privileged vs. vanilla teacher conditioning in OPD, and cross-lingual transfer between Japanese and Spanish (Table 5), where a monolingual judge trained only on Japanese transfers surprisingly well to Spanish (65.91% vs. 66.02% for the native Spanish judge).
Critical weaknesses
  • There appears to be a missing baseline that is critical for validating the core pipeline: training a student SFT model on uncalibrated (direct) Gemini 3.1 Pro labels vs. calibrated Gemini 3.1 Pro labels. Table 2 shows that calibrated hints improve the teacher's own dimensional macro-F1 from 42.3% to 48.5% (while slightly lowering its Overall accuracy from 71.3% to 69.9%), but in Sections 5 and 6 and Table 9, all student models are trained exclusively on the calibrated supervision. Without comparing a student trained on direct Gemini labels against one trained on hint-calibrated Gemini labels, it is unclear how much of the student's final performance actually comes from the select-calibrate-scale mechanism.
  • In Section 5 (Eq. on p. 7) and Appendix C.1, the dimensional reward r_dim in RL completely excludes dimensions whose reference verdict is TIE, even though TIE accounts for 55.1% of human dimension-level judgments. Because the policy receives positive reward only on decisive A/B dimensions and zero reward on TIE dimensions, RL naturally encourages the model to over-predict A/B and drift on ties, which directly explains why dimensional macro-F1 drops from 52.37% to 47.64% after RL. It is unclear why TIE was excluded from r_dim when matching TIE behavior was a primary motivation in Section 4.
  • Some training details across SFT, OPD, and RL make the comparisons slightly hard to interpret. First, in Figure 4, the authors keep the number of epochs fixed (6 epochs) while increasing the SFT data fraction from 20% to 100%, and conclude that "more supervision is not always helpful" because the 100% run drops in accuracy. Since the subsets are drawn from the same pool, the 80%-to-100% slice shouldn't be noisier than the rest; rather, the 100% run simply takes 25% more gradient steps and may just be overfitting. Second, Appendix C.1 notes that RL is run on only "651 labeled prompts" (compared to 7,840 for SFT and OPD), without explaining how these 651 prompts were selected or whether they overlap with the human dev set. Third, in Privileged OPD (Section 5), the paper states that the teacher receives an "evidence brief c derived from scalable supervision", but the main text does not clearly explain what c contains (e.g., whether c explicitly gives the teacher the Gemini target verdicts/rationales).
  • The claim in the abstract and Section 6.2 that RL improves the acoustic grounding of rationales rests on a very small human evaluation slice: only 28 paired judgments for SFT vs. SFT+RL (11 vs. 1, with 16 rated as no difference; Table 11), and Appendix E.2 notes that inter-rater repeat agreement on overall rationale preference is only 27%. While it is an interesting observation, highlighting it as a main takeaway feels a bit stretched given the sample size and low annotation stability.
Questions for authors
  1. Have you trained the 7B SFT student on the uncalibrated (direct) Gemini 3.1 Pro supervision from Table 2? How much does the select-calibrate-scale hint mechanism improve the trained student's Overall accuracy and dimensional macro-F1 compared to distilling from uncalibrated Gemini labels?
  2. Why were reference TIE labels excluded from the dimensional reward r_dim during RL training, and what happens to dimensional macro-F1 if matching a reference TIE is also rewarded? Also, what are the 651 labeled prompts used for RL in Appendix C.1, and why wasn't RL run on the same 7,840-pair pool used for SFT and OPD?
  3. In Privileged OPD (Section 5), what exact information is included in the "evidence brief c" passed to the teacher? Does c contain the soft metric hints, or does it contain the full verdicts and rationales generated by Gemini 3.1 Pro?
  4. In Figure 4, if you control for the total number of optimization steps (or select the best checkpoint along the training curve) rather than fixing training at 6 epochs, does using 100% of the supervision pool still underperform 80%?
3 comments
AuthorsOpenReview ↗

Author Response to 6LJz (1/2) Additional Experiments & Analyses

We thank the reviewer for identifying the most direct baseline for validating our calibration pipeline and for pointing out important training and data details that deserved clarification.

1. SFT with Uncalibrated Gemini Supervision (weakness 1 & question 1)

We agree that this is a direct baseline and have now added it. We trained SFT students using uncalibrated Gemini 3.1 Pro supervision without calibrated hints, on exactly the same audio comparisons as the original calibrated supervision. We kept the training configuration, seeds, and DEV-based checkpoint selection identical. Results are mean ± SD across three seeds. We also report 95% paired-bootstrap confidence intervals (20,000 resamples of the same 290 TEST comparisons), averaging seed-wise metrics within each condition before computing differences. Following suggestions from Reviewer F17h, we also added two simple baselines for comparison.

MethodOverall Acc. ↑Dim. Macro-F1 ↑Tie-rate MAE ↓
Acoustic-metrics-only model61.7238.7626.01
Human-DEV-only SFT62.07 ± 3.8948.86 ± 10.4814.00 ± 4.83
Uncalibrated Gemini SFT69.66 ± 1.1948.78 ± 0.9026.70 ± 1.71
Hint-calibrated Gemin SFT69.54 ± 0.8052.37 ± 0.1018.97 ± 0.44
Calibrated SFT − Uncalibrated SFT [95% CI]-0.11 [-4.25, 4.02]+3.59 [1.18, 5.91]-7.74 [-9.54, -5.72]

Calibration improves the trained student's dimensional macro-F1 by 3.59 points and reduces Tie-rate MAE by 7.74 points, with both confidence intervals excluding zero, with no detectable difference in Overall accuracy. The largest dimensional gains occur in Speaker (+10.94, CI [6.72, 15.02]) and Pronunciation/Accent (+7.46, CI [2.66, 12.08]) dimensions. These results confirm that the benefits of human calibration transfer from the machine labeler to the trained 7B student, improving alignment with human dimensional judgments rather than Overall accuracy. We have added the new baselines to Table 3 and summarized the student-level calibration benefits in the final paragraph of Section 4.3.

2. Tie-aware RL reward (Weakness 2, Q2)

Excluding reference Tie labels from the dimensional reward was an empirical design choice, not a principled restriction. We had explored a three-class reward (A/B/Tie), but did not pursue it further because its DEV Overall accuracy was approximately 2 points below the SFT reference. Prompted by the reviewer, we revisited this exploratory run and examined its TEST results:

ConfigurationOverall Acc. ↑Dim. Macro-F1 ↑A/B class-F1 ↑Tie F1 ↑Tie-rate MAE ↓
SFT69.5452.3753.5350.0318.97
Tie-exclusive RL71.8447.6451.3040.3427.30
Tie-inclusive RL67.9353.0452.9553.2416.29

The reviewer correctly identified an important trade-off overlooked by our Overall-based model selection. Compared with gated RL (tie-exclusive), the Tie-inclusive run achieves higher dimensional macro-F1 (47.64 → 53.04), including both A/B and Tie F1, and lower Tie-rate MAE (27.30 → 16.29), at the cost of lower Overall accuracy (71.84 → 67.93).

We appreciate the reviewer prompting us to revisit this result. We have added this exploratory run to Table 9 (Additional Experiments) and briefly discussed its Overall–dimensional trade-off in Section 5, highlighting Tie-inclusive rewards as an option when dimensional alignment is prioritized.

3. Supervision scaling: Non-monotonic gains persist under matched update budgets (Weakness 3, Q4)

We agree with the reviewer that fixing epochs introduces unequal optimization budgets across data fractions. We therefore re-ran the comparison with matched update counts across all five fractions within each seed: 600/700/600 updates for seeds 123/456/42, respectively, fixed from the original 80% runs. Values are Overall accuracy (%), mean ± SD across three seeds.

SupervisionDEV Acc. (%)TEST Acc. (%)
20%72.42 ± 1.3367.47 ± 1.44
40%71.57 ± 1.4668.51 ± 0.53
60%75.08 ± 1.6968.05 ± 1.39
80%77.00 ± 1.4669.54 ± 0.80
100%74.65 ± 1.1267.82 ± 0.53

First, the lower mean performance at 100% persists without additional optimization steps. With updates matched, 100% still scores below 80% on both DEV and TEST. Thus, this pattern does not require the additional updates introduced by the original fixed-epoch setup.

Second, we agree that the additional 20% should not be presumed noisier. Gemini labels are not human ground truth: more supervision may reinforce the labeler’s systematic biases rather than improve agreement with human judgments.

Our conclusion is that increasing machine-generated supervision does not automatically translate into better human alignment. We have updated Section 6.1 Q1 and Figure 4 (both plot and caption).

AuthorsOpenReview ↗

Author Response to 6LJz (2/2) Clarifications

4. RL data details (Q2)

The 651 RL examples are subset of the same machine-labeled training pool used for SFT and OPD, with no overlap with the human DEV or TEST sets. They were selected for high agreement between the uncalibrated and hint-calibrated Gemini supervision: the Overall and all five dimensional verdicts had to agree. We use this small subset over the full 7,840 example because RL refines an already trained judge, so we prioritized consistent reward targets over broader coverage. We have clarified the data details in Section 5, immediately before the reward definition.

5. OPD data details (Q3)

Yes, the evidence brief c contains Gemini-generated verdicts and rationales, reformatted into compact, teacher-only evidence. It does not contain soft acoustic-metric hints. This follows the privileged-information setting of OPSD (Zhao et. al., 2026), where the teacher conditions on a reference solution. We have clarified the exact contents of the teacher-only evidence brief c in Section 5.

6. RL improves rationale grounding as a directional finding (Weakness 4)

We retain the RL grounding result as an important directional empirical observation. We agree that 28 human judgments are insufficient to establish a robust, general improvement, and have therefore removed this claim from the abstract.

Nevertheless, the matched-verdict listening comparison reveals a clear directional pattern: listeners preferred RL rationales for audible-evidence grounding 11 to 1, with 16 ties. Importantly, the 11 preferences for RL came from six different raters. We hope this observation motivates further research through larger-scale human listening studies.

Dear reviewer 6LJz, we wanted to kindly follow up on our response. If our new baselines and additional analyses have addressed your concerns, we would be grateful if this could be reflected in your final assessment. We are happy to clarify anything further.

Reviewer_F17hOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies diagnostic speech judging, where a judge compares two target-language speech candidates against a source-language reference and returns an overall winner, a verdict on each of five perceptual dimensions (speaker, emotion, timing, pronunciation and accent, audio artifacts), and a rationale that cites audible evidence. The motivation is that existing speech judges reduce quality to a single naturalness score, while prompting a frontier audio-language model directly gives verdicts that disagree with humans at the dimension level, almost never choose Tie, and come with rationales that are not grounded in the audio. The central question is whether a few hundred human judgments are enough to build such a judge without collecting human labels at training scale.

The proposed pipeline, SpeechCritic, uses the human labels to calibrate machine supervision rather than to replace it. A small human-labeled development set is used to select, for each dimension, which off-the-shelf acoustic measurement agrees with human verdicts, and to map that measurement to probabilities over A, Tie and B. Those probabilities are then passed as non-binding hints to a frontier model, which still listens to the audio and produces verdicts and rationales for more than ten thousand unlabeled comparisons. The point of keeping the hints non-binding is that the measurements alone are weak, so the model should be able to override them, while the calibration should correct its systematic biases such as under-predicting Tie. A 7B student is then trained on this supervision, and the paper compares supervised fine-tuning, on-policy distillation and reinforcement learning not as a ladder but as signals that shape different behaviors: SFT to establish the output format and task, distillation to inherit the teacher’s judgment profile, and RL to sharpen whichever verdict behavior the reward encodes.

Validation rests on a held-out human test set that is never used for calibration or model selection. The hint pipeline is checked with a paired bootstrap over test comparisons, the training signals are compared over three seeds and stratified by human consensus, rationale quality is assessed by native listeners who hear the audio before reading the explanations, and the whole pipeline is repeated for a second target language and probed on an external dubbing benchmark.

Critical strengths

The core idea is sensible and cost efficient. Using a few hundred human labels to decide which acoustic measurements a machine labeler should trust, and then letting the labeler listen and override, is a better use of scarce expert annotation than either training directly on it or trusting the frontier model unaided. The calibration result is supported by a paired bootstrap that excludes zero for both the dimension-level gain.

The analysis of training signals is broad and mostly multi-seed, covering the amount of supervision, which modules to adapt, teacher choice and conditioning, student initialization, reward design and composed training. The paper also shows more supervision is not always better, persuasive rationales can accompany wrong verdicts and a text-only grader of rationales is uninformative.

Also, Judging speech along named perceptual dimensions with audible evidence, rather than a single preference score, is what a speech engineer actually needs to act on, and prior trained judges such as SpeechJudge and GSRM focus on naturalness or overall quality.

Critical weaknesses

One of the biggest weakness is that the training-signal narrative is not statistically supported by the test set. The English-Japanese test set has 290 comparisons, so the standard error of an accuracy near 70% is about 2.7 points. The contrasts that drive the paper’s story sit inside that range: SFT reaches 69.5% overall accuracy, gated RL 71.8%, privileged distillation 73.8%, and the composed recipe 73.9%. The seed standard deviations of around one point describe training variance, not sampling variance over test items, and the only paired bootstrap in the paper is for the calibration result. The claim that SFT, distillation and RL shape distinct judge behaviors, including the finding that RL gains concentrate on high-consensus cases and that distillation lowers dimensional macro-F1, therefore rests on differences that could plausibly be noise.

Related to this, the paper does not include the baselines that would show what the pipeline adds. The question it poses is whether limited human judgments can replace training-scale labels, and more reasonable comparator would be to train directly on the roughly 300 human labels with or without the machine pool and see how far that gets. A second natural baseline is a judge built from the calibrated acoustic probabilities alone, with no language model, since those measurements already reach macro-F1 between 29 and 55 per dimension on the development set. Neither is reported, so the reader cannot tell whether the ten thousand comparison machine pool is doing the work or whether the small human set and a few acoustic features would do nearly as well. Existing trained judges are cited but not run on the test set either.

A second concern is how much of the measured gain is diagnostic rather than a Tie-rate effect, and whether it generalizes beyond the constructed pools. The test set is Tie-heavy, with human-majority Tie rates between 52% and 88% depending on the dimension, and the hint pipeline’s largest effect is on Tie behavior, cutting the Tie mismatch by 10.4 points while overall accuracy does not move. On such a distribution, matching the Tie marginal alone raises three-class macro-F1, so the 6.3-point dimension-level gain could come largely from predicting Tie more often rather than from better comparison-level judgments. The paper acknowledges that marginal Tie prevalence does not imply correctness but does not run the control that would separate the two. On generalization, all candidates are synthesized from two TTS families with pairs drawn randomly within a target dimension, and the only external check is the VOX-DUB dubbing benchmark, where the trained judges reach mean macro-F1 of 38.7 and 38.8 against 40.0 for zero-shot Gemini and 57.0 for the human lower reference, and no model beats an always-Tie baseline on the voice dimension. The paper describes this as supporting transfer. It reads more naturally as evidence that the gains are specific to the constructed pools and that the trained 7B judge is, out of distribution, no better than the frontier model it was meant to improve on.

Finally, the claim that RL improves rationale grounding rests on very little. The comparison involves 28 human judgments, with 11 favoring the RL model, one favoring SFT and 16 finding no difference, and the sign test is computed on the 12 directional cases. Repeat agreement among the raters is 27% for overall rationale preference. The paper’s own analysis also shows that raters prefer the rationales of a distillation teacher that produces wrong verdicts, which means human rationale preference is not a reliable signal of grounding. The abstract’s statement that listeners find RL rationales cite more specific cues is stronger than this evidence supports.

Questions for authors
  • Could you report paired bootstrap intervals over test comparisons for the main contrasts in the training-signal table in particular SFT versus gated RL, SFT versus privileged distillation and the composed recipe versus each of its parts?
  • What does supervised fine-tuning on the roughly 300 human development labels alone achieve and what does a judge built only from the calibrated acoustic probabilities achieve on the same test set?
  • How much of the 6.3-point dimension-level gain from hints remains if Tie is excluded from the gold labels, or if Tie is injected at random at the calibrated rate as a control?
  • Given that the trained judges score below zero-shot Gemini on VOX-DUB and no model beats always-Tie on voice, why should that result be read as positive transfer rather than as no advantage out of distribution?
  • Why does the Spanish-trained judge score higher on the Japanese test set than the Japanese-trained judge and does this suggest the judge is learning format and language-independent acoustic cues rather than language-specific perception?
  • Will the human benchmark and the machine-labeled comparison pool be released given that the frontier labeler is proprietary and will change?
5 comments
AuthorsOpenReview ↗

Author Response to F17h (1/3) Additional Analyses

We sincerely appreciate the reviewer’s careful attention to statistical rigor and their suggestion of two simple baselines to examine the added value of our select–calibrate–scale pipeline. We address the concerns as below:

1. Paired bootstrap intervals for training-signal (Weakness 1, Q1)

Following the reviewer’s suggestion, we computed 95% paired-bootstrap confidence intervals over the same 290 English–Japanese test comparisons (20,000 resamples) for Table 3. In each resample, we sampled 290 comparisons with replacement, using the same sampled items across methods, and computed the metric differences after averaging over three fixed training seeds. A 95% CI excluding zero indicates a statistically significant difference at the 5% level. As shown below, the key training-signal contrasts remain statistically supported after accounting for test-item uncertainty.

ContrastOverall Acc.High Acc.Low Acc.Dim. Macro-F1
Gated RL − SFT+2.30 [0.00, 4.60]+3.92 [1.42, 6.63]0.00 [−4.10, 4.17]−4.72 [−6.64, −2.76]
Privileged OPD − SFT+4.25 [0.57, 7.93]+5.49 [1.34, 9.89]+2.50 [−3.88, 8.87]−7.76 [−9.92, −5.49]
Composed − Gated RL+2.07 [−1.84, 5.98]+0.20 [−4.25, 4.62]+4.72 [−2.31, 11.77]+5.23 [2.12, 8.26]
Composed − Privileged OPD+0.11 [−3.22, 3.56]−1.37 [−5.02, 2.03]+2.22 [−4.26, 8.87]+8.26 [5.41, 10.97]
Composed − SFT+4.37 [0.57, 8.16]+4.12 [−0.40, 8.70]+4.72 [−1.65, 11.21]+0.51 [−2.44, 3.45]

Compared with SFT, gated RL improves high-consensus accuracy (+3.92 points, CI [1.42, 6.63]) while reducing dimensional macro-F1 (−4.72, CI [−6.64, −2.76]). Privileged OPD improves Overall accuracy (+4.25, CI [0.57, 7.93]) while also reducing dimensional macro-F1 (−7.76, CI [−9.92, −5.49]). The composed training model improves Overall accuracy over SFT (+4.37, CI [0.57, 8.16]) and dimensional macro-F1 over both gated RL (+5.23, CI [2.12, 8.26]) and privileged OPD (+8.26, CI [5.41, 10.97]). Its Overall accuracy, however, is not statistically distinguishable from either component method.

These results support our central conclusion that SFT, OPD, and RL produce distinct judging behaviors rather than forming a simple accuracy ladder. The key contrasts are not explained by test-item sampling noise alone. We have updated Section 5 and Appendix D.1.

2. Dimensional-level Gain vs Tie-rate effect (Weakness 3, Q3)

First, we emphasize that Tie is itself an essential diagnostic judgment. When two candidates have no audible difference along a perceptual dimension, forcing an A/B verdict invents a distinction and can encourage an ungrounded explanation. Recognizing when not to assert a difference is therefore part of alignment with human perception, motivating our inclusion of Tie F1 and Tie-rate MAE as primary metrics.

We thank the reviewer for suggesting this analysis. We agree that the dimensional gain should be broken down and tested against a control that separates better comparison-level judgments from simply predicting Tie more frequently.

Following this suggestion, we randomly replaced uncalibrated Gemini’s A/B predictions with Tie to match the hinted labeler’s Tie count within each dimension. This recovers only 1.45 points of the 6.25-point gain: macro-F1 increases from 42.28 to 43.73, compared with 48.53 using calibrated hints. At the same total of 404 Tie predictions, random matching produces approximately 323 correct Ties and 81 false Tie calls, whereas calibrated hints produce 351 and 53, respectively. Decomposing the actual gain, 2.22 points come from the A/B class-F1 terms and 4.03 from the Tie term.

These analyses clarify that the improvement is not simply a consequence of predicting Tie more frequently; calibration improves where the model predicts Tie. We have incorporated this control and the gain decomposition into Section 4.3 of the revised manuscript.

AuthorsOpenReview ↗

Author Response to F17h (2/3) Additional Experiments

3. Two additional simple baselines (Weakness 2, Q2)

To assess what our proposed “select–calibrate–scale” pipeline adds, we evaluated two additional baselines as suggested by the reviewer. We have added these results to Table 3.

  • Human-DEV-only SFT: We fine-tune Qwen2.5-Omni-7B on 232 human-labeled DEV comparisons (83 comparisons without majority label for at least one dimension were excluded) across three seeds, without any Gemini-generated supervision. Because the human annotations contain no rationales, we train this baseline to predict verdicts only.
  • Acoustic-metrics-only model: Using human DEV labels, we calibrate five multinomial logistic regressions (as described in Section 4.2), one per dimension, to predict TEST dimensional verdicts from signed A–B acoustic metric gaps. We train a separate binary logistic regression on all five gaps to predict Overall preference. This simple machine-learning baseline involves no LLM fine-tuning and produces verdicts only.
MethodOutputs Rationale?Overall Acc. ↑Dim. F1 ↑Tie MAE ↓
Zero-shot Qwen2.5, verdict-onlyNo10.6925.2041.55
Acoustic-metrics-only modelNo61.7238.7626.01
Human-DEV-only SFTNo62.07 ± 3.8948.86 ± 10.4814.00 ± 4.83
SpeechCritic SFTYes69.54 ± 0.8052.37 ± 0.1018.97 ± 0.44

These baselines clarify what SpeechCritic adds. Compared with human-DEV-only SFT, it improves Overall accuracy by 7.47 points (95% CI [2.07, 12.76]) and dimensional macro-F1 by 3.51 points ([0.31, 6.71]), with lower cross-seed variability. This supports amplifying limited human judgments into scalable verdict-and-rationale supervision. Against the acoustic-metrics-only baseline, the corresponding differences are +7.82 points in Overall accuracy (95% CI [1.61, 14.14]) and +13.61 points in dimensional macro-F1 ([9.83, 17.25]), suggesting that acoustic metrics should inform an audio-language judge rather than replace it.

Beyond Overall accuracy, the per-dimension macro-F1 results show where SpeechCritic adds value over the two simple baselines. SpeechCritic achieves the highest mean score in four of five dimensions, while human-DEV-only SFT performs best on Accent (updated in Appendix D.2).

Method / MetricSpeakerEmotionTimingAccentArtifacts
Acoustic-metrics-only model42.00 (spk sim)22.45 (arousal)42.31 (duration ratio+DTW)55.85 (CER)31.18 (UTMOS)
Human-DEV-only SFT46.99 ± 9.1043.08 ± 9.0840.26 ± 9.1362.46 ± 7.8351.51 ± 18.25
SpeechCritic SFT51.58 ± 2.8344.10 ± 0.9644.57 ± 2.6359.42 ± 4.1062.15 ± 2.62

Acoustic metrics capture useful perceptual information, but SpeechCritic achieves stronger dimensional performance by using them as calibrated hints alongside the audio. Again, these measurements should therefore inform an audio-language judge rather than replace it.

AuthorsOpenReview ↗

Author Response to F17h (3/3) Clarifications

4.VOX-DUB Transfer (Weekness 3, Q4)

We thank the reviewer for raising this point, and we agree that our trained 7B judges do not outperform Gemini overall on VOX-DUB. We would like to clarify that our transfer claim is more limited: the training pipeline develops dimension-level judging ability that carries over to an external benchmark with different synthesis systems and an only approximately aligned evaluation rubric.

To support this, we compare against the untrained Qwen2.5-Omni-7B, which achieves 27.4 mean macro-F1 on VOX-DUB. Our English–Spanish and joint-language judges reach 38.7 and 38.8, respectively. We apologize for omitting this baseline from Table 12 and have added it in the revised version. Both trained judges also outperform zero-shot Gemini on Pronunciation (53.8/51.6 vs. 45.0), although Gemini retains a higher mean macro-F1 overall (40.0). The Voice dimension remains challenging for all evaluated models, including Gemini: none exceeds the always-Tie baseline of 29.5. We have revised Appendix D.3 text to make the scope of this claim clearer.

ModelMean macro-F1Pronunciation F1Voice F1
Untrained Qwen2.5-Omni-7B27.428.324.2
SpeechCritic (ES)38.753.828.7
Zero-shot Gemini 3.1 Pro40.045.026.1

5.Why Spanish-trained judge scores higher on Japanese test set (Q5)

Our five-dimensional rubric primarily evaluates how speech sounds, focusing on its acoustic aspects. In addition, transcripts are provided directly to the judge in our setup. Together, these design choices prevent the judge's evaluation from relying solely on language-dependent semantic understanding.

One explanation is that broader, more diverse Spanish coverage in our internal data may expose the judge to richer acoustic variation, supporting transfer to Japanese. This motivates exploring multilingual supervision beyond the evaluation language. We discuss these implications in Section 6.3.

6. RL improves rationale grounding as a directional finding (Weakness 4)

We retain the RL grounding result as an important directional empirical observation. We agree that 28 human judgments are insufficient to establish a robust, general improvement, and have therefore removed this claim from the abstract.

Nevertheless, the matched-verdict listening comparison reveals a clear directional pattern: listeners preferred RL rationales for audible-evidence grounding 11 to 1, with 16 ties. Importantly, the 11 preferences for RL came from six different raters. We hope this observation motivates further research through larger-scale human listening studies.

7. Report existing specialized speech judges (Weakness 2)

Existing specialized judges (e.g., SpeechJudge for naturalness and QualiSpeech for low-level quality) target different evaluation objectives from our reference-based cross-lingual comparison, with different task-specific input formats, system prompts, rubrics, and output formats. In our preliminary attempts to apply them to our task, they often fail to produce the required five-dimensional verdicts and evidence-grounded rationales, preventing a directly comparable evaluation.

8. Data release (Q6)

Because the source audio is copyrighted, we do not currently plan to release the audio pairs, the human benchmark, or the machine-labeled comparison pool.

To support reproducibility, our supplementary materials include the full human-preference calibration and model-training code, enabling researchers to instantiate SpeechCritic on their own speech/audio evaluation data, even as proprietary frontier labelers evolve.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers and Area Chair

We thank all three reviewers for their constructive feedback and for recognizing two central contributions of our work: (1) the well-defined, resource-efficient select–calibrate–scale pipeline, and (2) the transparent analysis of training signals, including positive and negative findings, their trade-offs, and practical implications.

Following their valuable suggestions, we have conducted additional experiments and analyses and updated the manuscript accordingly:

  • Statistical validation (F17h). We added paired-bootstrap confidence intervals, providing statistical support for our central finding that different training signals produce distinct judging behaviors.
  • Three additional baselines (F17h, 6LJz, XmEE). We evaluated human-DEV-only SFT, an acoustic-metrics-only model, and SFT on uncalibrated Gemini supervision, strengthening the evidence for SpeechCritic’s added value in building human-aligned diagnostic judges.
  • Deeper analysis of Tie behavior (F17h, 6LJz). We added a random Tie-frequency control and revisited Tie-inclusive RL, showing that diagnostic alignment depends on making the right judgments—not simply predicting Tie more often or maximizing Overall accuracy.
  • Clearer presentation and practical guidance (F17h, 6LJz, XmEE). We clarified experimental details and refined the scope of our claims, making the revised paper more transparent, reproducible, and actionable for researchers building diagnostic judges.

We hope these additions address the reviewers’ concerns and make both the empirical evidence and the practical guidance of SpeechCritic clearer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.