acceptodds
Under review as a conference paper at ICLR 2027

AudioCaliper: A Controlled Probe of Audio-LLMs as Speech Quality Judges

Abstract

Audio large language models (audio-LLMs) now support reasoning beyond simple classification. They are increasingly used to train other models. They curate speech data, score text-to-speech outputs, gate deployments, and post-train generators as reward models. However, whether they can be trusted in these roles is rarely tested. In this paper, we present AudioCaliper, an open-source, reusable framework for analyzing how an audio-LLM behaves as a quality evaluator. For each clip, we vary one factor at a time while keeping the rest fixed, either the acoustic degradation, the presence of speech, a quality claim in the text prompt or the answer position. This setup allows us to attribute changes in the output to the manipulated factor. We apply AudioCaliper to LibriSpeech and VCTK using Qwen2-Audio, Qwen2.5-Omni, SQ-LLM, SpeechJudge-GRM, SALMONN-SQA and QualiSpeech. Our results reveal each judge's sensitivity to the audio input and text prompt. Fine-tuned judges reach an acoustic sensitivity of 0.87, while zero-shot judges near zero. However, fine-tuning does not eliminate other failures. Three out of six judges, two of them fine-tuned, cannot separate speech from silence, and two score silence higher than real speech. All six judges remain sensitive to quality claims in text prompts. Describing a clip as high quality rather than low quality moves a fine-tuned judge's score by 31% of its rating scale and a zero-shot judge's by 86%. In pairwise comparisons, five out of six judges rely on the position of the clip rather than the input audio. Our findings call for evaluating a judge on the decisions it will support, whether scoring, ranking or rewarding, before its scores are used to train or evaluate other models.

Primary area:
interpretability and explainable AI
Keywords:
speech quality assessment, audio language models, LLM-as-judge, evaluation, trustworthy AI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_emZJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether audio-language models used as speech-quality judges respond to acoustic evidence, textual quality claims, or presentation order. It evaluates six released models using controlled interventions derived from 200 LibriSpeech utterances, with selected experiments repeated on 195 VCTK utterances. The experiments cover graded acoustic degradation, replacement with silence or noise, conflicting textual claims, and order-swapped pairwise comparisons. The paper reports that quality fine-tuning improves degradation sensitivity but does not consistently eliminate prompt sensitivity, inappropriate scores for silence, or failures under concatenated comparison. It also introduces a text-equivalent SNR statistic intended to express textual influence in acoustic units.

Critical strengths
  1. The matched interventions expose failures that aggregate agreement can miss. The same utterances are evaluated across acoustic and textual conditions, reducing confounding from speaker and recording differences. Tables 3 and 6 show that a model can respond strongly to degradation while remaining susceptible to a quality claim: SQ-LLM has a white-noise ASI of 0.77, yet its clean-audio score changes by 1.23 points between the two claims. This is a useful distinction for selecting and auditing speech evaluators.

  2. The paper checks several plausible explanations for its main findings. Recorded noise supplements synthetic white noise, selected effects reproduce on VCTK, and Appendix F tests alternative claim wordings. Appendix C also resamples utterances together with their derived conditions rather than treating all scores as independent observations. These choices strengthen the evidence that the reported acoustic and textual effects are not confined to one wording or one corpus.

Critical weaknesses
  1. The pairwise experiment does not adequately distinguish positional preference from failure to interpret the comparison format. Five judges receive one concatenated waveform with a 0.5-second silent separator. Appendix H explicitly recognizes that a judge may fail to locate the boundary, but scoring the clips separately does not resolve that possibility. It establishes useful single-clip discrimination, not successful identification of both candidates during comparison. In section 5.4, the authors appropriately attributes some failures to the interface, but the stronger interpretation as position bias remains underdetermined. This matters particularly for SQ-LLM. The pointwise comparison also uses a different denominator. In Table 15, SQ-LLM’s 0.82 accuracy applies only to the 48% of pairs with unequal scores, whereas its 0.54 pairwise accuracy covers all trials. Assigning half credit to pointwise ties gives approximately 0.48×0.82+0.52×0.5=0.6540.48\times0.82+0.52\times0.5=0.654, so the comparable improvement is about 0.114 rather than the displayed 0.28. The improvement survives, but the reported gap overstates the practical recovery.

  2. TES is not sufficiently justified as an equivalent acoustic shift. Section 4.3 specifies noisy conditions from −20 to 20 dB plus clean speech, yet Table 11 reports fitting ranges extending to 30 dB and a total span of 50 dB. Figure 3 correctly notes that clean speech has no finite SNR. The manuscript therefore appears to assign the clean condition an artificial finite coordinate without explaining or testing that choice, which can change the fitted slopes and TES. Additionally, Eq. 4 divides an intercept difference by the average slope even when the two slopes differ substantially. For QualiSpeech, Table 11 reports 0.0830 and 0.0399 points/dB. Such lines do not define a single horizontal separation: the equivalent shift depends on the score being matched. The paper acknowledges that TES is approximate, but does not quantify the approximation error or demonstrate matched-score support for the reported shifts. This weakens the interpretation of values such as 32.5 dB as degradation that a textual claim can compensate for.

Questions for authors
  1. Was SQ-LLM evaluated using its released comparison format, and is there evidence that the concatenated-input judges identify both candidate segments? A supported-interface result or a direct boundary-recognition control would materially change my assessment of the position-bias claim. Please also compare pointwise and pairwise performance on a common denominator, with an explicit tie policy.

  2. Please clarify the coordinate assignment and whether the reported values remain stable when clean speech is excluded. For unequal slopes, what matched-score range supports the claimed equivalent shifts without extrapolation? Stability under these checks would substantially strengthen the proposed metric.

4 comments
AuthorsOpenReview ↗

To Reviewer emZJ,

We thank the reviewer for the precise technical comments. You wrote that a supported-interface result or a boundary-recognition control would materially change your assessment of the position-bias claim. We ran both, along with the two analyses you asked for. The detail is in parts 1 to 3 of our response to all reviewers.

W1, position preference against failure to interpret the format. Section 5.4 already separates two readings of the concatenated result. It assigns SQ-LLM, SALMONN-SQA and QualiSpeech to the reading in which the judge can order the recordings and the interface loses that ordering. Table 7 supports this. Every pair is judged in both orders, and SQ-LLM picks the first slot both times on 0.919 of pairs, with a position-immune discrimination of 0.08, so its answer follows the slot and not the recording. You asked for direct evidence, and the two new controls provide it.

On its released two-audio interface SQ-LLM reaches 0.989 accuracy and names slot A on 0.511 of trials, against 0.54 and 0.96 concatenated, with every reply decidable and an identical decision rate under both slot orders. In the identification control SQ-LLM reports two clips on 0.967 of trials and names the degraded half on 0.911 with a near-balanced slot figure, yet answers the preference question from position on 0.872. Its perception of the two clips is intact and does not reach its comparative answer. Raising the separator from 0.5 to 2.0 seconds moved that figure only to 0.844, so separator length was not the obstacle. SALMONN-SQA reports three clips on 98 of 180 trials, so for that judge the presentation does prevent segmentation. Both tables are in part 2.

The revision will add both results to Section 5.4, keep the two causes apart, and qualify the abstract's statement that five of six judges rely on position. The numbers of the other four concatenated judges do not change, since they accept a single audio input.

On the denominator, our recomputation matches your figure of 0.654, which gives an improvement of 0.11 over all pairs. As you noted, the improvement remains. The full accounting is in part 1.

W2, the justification for TES. The paper presents TES as an approximation, and your two checks let us quantify it.

On the coordinate, clean speech was placed at 30 dB so that it could enter the regression, and the revision will state this. The refit on the nine measured noise levels is in part 1 and leaves the ordering of the judges unchanged.

On unequal slopes, the slope ratios and matched-score ranges are in part 1. Where the slopes are close a single value is well defined, and SQ-LLM's matched range spans 0.3 dB, from 31.8 to 32.1. Where they diverge, as for QualiSpeech with a ratio of 2.08, we report the range of 12.9 to 23.0 dB in place of a single value.

The revision will define the headline figure as the horizontal separation at a fixed reference score of 3.0, so one value is always recoverable without extrapolation, and will report the matched range and slope ratio beside it. The statement that a textual claim can offset tens of decibels holds for SQ-LLM. You wrote that stability under these checks would substantially strengthen the metric, and the ordering of the judges is stable under both.

Thank you for the additional experiments. The native-interface and boundary-recognition results substantially address my first concern, and the common-denominator analysis resolves the comparison issue. My remaining concern is regarding TES: fixing the reference score at 3.0 does not itself guarantee a comparison without extrapolation. Please report the SNR implied by each fitted line at this score, verify that both lie within the retained finite-SNR support, and clarify how the finite-only TES values relate to the matched ranges. I am retaining my current rating for now, but would consider move to 3 if this support is established or the corresponding claims are appropriately narrowed.

Thank you, reviewer emZJ, for your response and time for checking our comments. We are grateful that you are considering the move to 3. We promise that we will try our absolute best to meet your additional comments. We are beginning to prepare additional analysis. Let us come back with the results and either 1) supplement the results or 2) narrow our claims, depending on the results, which we will share either way.

We would like to share one information, which is purely about timing. As far as we are concerned, "Author-Reviewer Discussion Period" ends 11/18 (tomorrow). While we are starting this additional analysis right away, we might not be able to come back within the next 24 hours. We will comeback with results and either supplement or narrow our claim on TES. In the meantime, we would greatly appreciate if reviewer could consider moving the score to 3 given the current circumstances (other concerns have been resolved and our claims regarding TES could be either supplemented or narrowed based on additional analysis). Thank you.

Thank you for the follow-up. You are right about the reference score, and the check below confirms it. We also think the check sharpens the paper's claim rather than weakening it and we explain below

The SNR implied at 3.0. Values are in decibels. GRM uses a 1 to 10 scale.

JudgeRetained SNR supportStudio line at 3.0Phone line at 3.0Both inside support
SQ-LLM-5 to 20-0.229.0No
GRM-10 to 20-9.9-1.4Yes, at the edge
SALMONN-SQA-15 to 209.120.2Marginal
QualiSpeech0 to 208.632.9No

Only GRM has both values inside the support, and its studio value sits at the lower edge. A fixed score of 3.0 therefore does not avoid extrapolation, and we drop it.

Finite-only values and matched ranges. Our earlier table mixed two fits, and we apologise. The matched ranges came from the published fits, which include the clean point at 30 dB. The finite-only column came from refits without it. On the finite-only fits, restricted to scores both lines reach inside the support, the ranges are 8.4 to 14.2 for GRM, 12.1 to 16.7 for QualiSpeech and -4.0 to 11.1 for SALMONN-SQA. The Eq. 4 value equals the horizontal separation only for parallel lines, so it need not fall inside these ranges.

Why SQ-LLM has no in-support value. No score is reached by both SQ-LLM lines inside the support. Studio-labelled audio at -5 dB scores 2.69 on average. Phone-labelled audio at +20 dB scores 2.58. The label effect spans the whole acoustic range we measured. Extrapolation was needed only to say how far beyond it the effect reaches. The failed check reflects the limits of our SNR range. It is not evidence against the effect.

The claim does not depend on the conversion. TES converts a measured score gap into decibels. The paper's argument rests on the gap itself, which is the text effect set against the acoustic effect on the same clips. That comparison needs no fitted line. The table divides the mean score gap between the studio and phone labels by the score change across the retained SNR range. Brackets are bootstrap intervals over the 200 clips.

JudgeLabel gapAcoustic change under the labelsRatioRatio against neutral
SQ-LLM1.461.351.09 [1.00, 1.19]0.64 [0.61, 0.68]
QualiSpeech1.132.080.55 [0.51, 0.58]0.48 [0.45, 0.51]
GRM0.932.750.34 [0.30, 0.38]0.43 [0.37, 0.50]
SALMONN-SQA0.252.480.10 [0.09, 0.11]0.10 [0.09, 0.11]

The fourth column uses the larger of the two label curves' changes, which is the conservative choice. The last column uses the change under the neutral prompt instead. For SQ-LLM the label moves the score as far as the full 25 dB of added noise does. The two labels also shrink SQ-LLM's response to noise, from 2.28 points under the neutral prompt to 1.35 and 1.25. So the text weakens the judge's response to the audio as well as shifting the score. SQ-LLM is the largest and SALMONN-SQA the smallest under both columns. QualiSpeech and GRM overlap under the neutral column, and the revision does not order them.

Decibel figures that stay in support. Against the neutral prompt, the studio label alone is worth about 15 dB for SQ-LLM, with a bootstrap interval of 13.9 to 16.1. This comes from the three lowest retained levels, where a match exists. SQ-LLM also scores clean speech labelled as a phone recording like studio-labelled speech at -1.7 dB. That figure uses the clean condition without assigning it a coordinate.

The revision. The headline measure becomes the ratio above, which needs no extrapolation. Decibel figures are reported only where both curves share a score inside the measured range, with the matched range and slope ratio beside them. For SQ-LLM the paper states that the label effect spans the full 25 dB range we tested. It reports 27.8 as an extrapolation beyond that range. The conclusion stands. A text claim moves SQ-LLM's score as far as 25 dB of real noise does. All numbers come from scripts/tes_support_check.py in the released code. We hope this narrows the TES claims in the way you described.

Reviewer_yK9SOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AudioCaliper studies whether audio language models judge speech quality based on the recording itself or are influenced by textual descriptions and presentation order. The authors evaluate six models by varying acoustic degradation, replacing speech with silence or noise, changing quality descriptions, and swapping the order of audio pairs. The main experiments use LibriSpeech, with selected tests repeated on VCTK. The results show that sensitivity to acoustic degradation does not guarantee reliable scoring: some models give silence high scores, follow misleading descriptions, or consistently select the same position in pairwise comparisons. The paper also examines alternative prompts and the role of reference transcripts.

Critical strengths
  1. The research question deserves attention. Audio judges may be used to select training data, compare models, or provide training rewards. If a judge follows a flattering description rather than the recording itself, it may favor the wrong outputs. Therefore, even if it agrees with human ratings on a standard test set, it remains important to examine what actually influences its scores.

  2. The experiments are convincing and reveal the effects of text and audio separately. Keeping the recording fixed and changing only its description directly tests whether text affects the score. Crossing clean/noisy recordings with positive/negative descriptions further tests whether a favorable description can make a noisy recording score higher than a clean one.

Critical weaknesses
  1. The paper needs to explain more clearly what it adds to the closest prior work. The central question is whether audio judges listen to the recording or follow text and presentation order. However, the cited study by Park et al. already examines this question by changing textual cues, swapping audio order, and comparing input formats. Describing earlier work as observations on natural data and suggesting that this question has not been asked before therefore overstates the novelty. AudioCaliper still provides useful additions, including tests of specialized quality judges, finer-grained distortion levels, and speech removal. The authors should explain what new insights these additions offer beyond the earlier findings.

  2. The experiments demonstrate risks in using these judges, but do not yet establish their suitability for the practical uses recommended in the paper. Table 4 shows that averaging scores across recordings within each constructed distortion condition improves agreement with PESQ. The paper uses this result to suggest that these judges can compare systems if scores are averaged over enough utterances. However, the experiments do not actually compare multiple speech synthesis or enhancement systems, nor do they check system rankings against human ratings. Knowing that a recording with more noise is usually worse is not enough to determine which of two real systems is better; their differences may involve pronunciation, prosody, naturalness, or synthesis artifacts. I do not think every basic check requires new human annotations. What is missing is evidence supporting the broader practical uses.

  3. Some failures are clear, but their causes remain uncertain. Giving silence a high score shows that the current scoring procedure does not properly handle recordings without speech. It does not necessarily mean that the model cannot recognize silence: the model may treat the absence of noise as a positive feature, or may not know that it should answer “not applicable.” Similarly, always choosing a fixed position after two recordings are concatenated shows that this comparison procedure is unreliable. It does not establish whether the model failed to separate the two recordings, misunderstood the task, or heard both correctly but still favored a particular position. Different causes require different solutions. The separate-scoring control is useful, and the authors acknowledge the limitations of the input format, but this distinction should also be maintained in the conclusions. To explain the cause, testing different input formats or ways of separating the two recordings would be more informative than reporting another overall accuracy score.

  4. Formatting issues. Citation formatting is problematic throughout the manuscript. For example, all citations on the first page of the Introduction are incorrectly formatted. In addition, the font or spacing appears to have been modified, making the text look more tightly packed.

Questions for authors

None

3 comments
AuthorsOpenReview ↗

To reviewer yK9S,

We thank the reviewer. The new experimental results are in part 2 of our response to all reviewers. This reply covers your four points.

W1, the relation to Park et al. We will rewrite the passage. Park et al. (arXiv 2607.13477) do intervene, through feature-blueprint judging, reference-conditioned judging, order swaps and injected incorrect specialist labels, and the revision will describe their work that way.

Our additions beyond that work are three. Park et al. target general audio-LLMs on tasks carrying ground-truth labels. We target judges fine-tuned for quality assessment, where no ground truth exists and the judge's score is itself the product. We vary degradation in graded steps, which is what makes a sensitivity coefficient and a decibel-denominated metric possible. And we include speech removal, which to our knowledge no prior work tests.

W2, the system-level recommendation. The paper is a controlled probe of failure modes. It does not set out to validate these judges for comparing systems, and one sentence built on Table 4 reached further than that. Table 4 supports a narrower statement, that per-condition averaging reduces per-clip noise on the degradations we constructed. Table 4 measures that agreement against PESQ. The paper also scores the same clips with DNSMOS P.808, a predictor trained on human ratings (Tables 3 and 5), and we have now repeated the Table 4 analysis against it. For the four judges of Table 4, averaging over the 20 conditions raises the rank correlation from between 0.20 and 0.72 per clip to between 0.80 and 0.91, so the statement does not rest on PESQ alone. The revision will limit the recommendation to that statement, and it will name the ranking of real synthesis or enhancement systems as outside the scope of this probe.

W3, the causes of the failures. The paper's claim is about the scoring procedure, as your first sentence on silence puts it. We agree that different causes need different remedies, and we can now separate them.

First, Appendix I already compares two instructions, and they fail differently.

InstructionText powerAudio power
Baseline+3.76+0.08
Disregard the claim+1.03-0.07
Describe before rating+1.42+1.12

Those are Qwen2.5-Omni. Telling a judge to disregard the claim lowers its reliance on the text without making it attend to the audio. Telling it to describe what it hears raises the audio power. Neither helps the fine-tuned judges, and describe-then-rate raises SQ-LLM's reliance on the text from +1.02 to +1.94. We will move this into the main text.

Second, on position, we ran the format manipulations you recommended. The results are in part 2. Each of your three accounts applies, to a different judge. SQ-LLM identifies both halves and still favours a slot. SALMONN-SQA cannot count the two clips. Qwen2-Audio counts the clips correctly but gives one letter on every comparison, so its failure is in responding and not in hearing.

Third, on silence, the prompt asks for a number only, which is the output these judges are used for. The replies therefore do not show whether a judge noticed the absence of speech. The free-text probe in part 3 separates your three accounts.

W4, formatting. The first page uses textual citation commands where parenthetical ones are required, and the same occurs later. All citations will be corrected.

The font and spacing are those of the template. The submission is US Letter at 612 by 792 points, uses the standard Type 1 font set, and carries 54 numbered lines per page, all matching the unmodified template. The dense appearance comes from long unbroken paragraphs in Sections 5 and 6, and we will split those.

Thank the authors for their rebuttal. I will increase my score.

Thank you, reviewer yK9S, for your reconsideration and taking time to check our rebuttal. Please let us know if you have any further comments that we can reflect and head towards 4. We will try our best to conduct all necessary supplementations.

Reviewer_dsYtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AudioCaliper, a controlled protocol for examining how audio-language models behave as speech-quality judges. Using matched versions of the same utterances, the experiments vary acoustic degradation, speech presence, textual quality claims, and presentation order. Six released judges are evaluated on LibriSpeech, with selected experiments repeated on VCTK. The results show that sensitivity to acoustic degradation does not guarantee robustness to misleading textual claims, silence, or changes in the comparison interface. Additional experiments examine prompt wording, pointwise versus pairwise evaluation, prompt-level defenses, and reference transcripts. The main contribution is a systematic empirical diagnosis of when these judges’ outputs reflect the audio and when they are influenced by other inputs.

Critical strengths
  1. The controlled design makes the results more informative than aggregate agreement scores alone. Holding the utterance fixed while manipulating acoustic conditions or textual claims separates effects that ordinary correlation-based evaluation can obscure. The finding that a judge can track acoustic degradation while remaining sensitive to misleading quality claims is particularly relevant to using these models as evaluators or reward models.

  2. The paper investigates its findings through several useful controls. Caption paraphrases, pointwise scoring, prompt-level defenses, and reference-transcript interventions help characterize the observed failures. Replicating the white-noise sensitivity and audio–text conflict results on VCTK also strengthens the evidence that these effects are not specific to LibriSpeech.

Critical weaknesses
  1. The pairwise findings require a more precise account of the models’ supported interfaces. The paper appropriately acknowledges that concatenation can cause failures, but it also describes GRM as the only judge with a native two-input path. The comparison template in SpeechLLM-as-Judges (Appendix B.1) uses two separately supplied audio inputs. Please clarify whether the evaluated SQ-LLM checkpoint supports this interface and why concatenation was used instead. This clarification matters for assessing how representative the SQ-LLM comparison results are of its intended use. The concern is limited to the pairwise evaluation and does not undermine the separate acoustic and textual intervention results.

  2. Two quantitative comparisons need clarification. First, Section 4.3 describes finite SNR levels from −20 to 20 dB plus clean speech, whereas Table 11 reports fitted ranges extending to 30 dB. Since clean speech has no finite added-noise SNR, its treatment matters for interpreting TES in decibels. Second, Table 15 compares unconditional pairwise accuracy with pointwise accuracy conditional on unequal scores. The disclosed tie rates vary substantially across judges, so the reported gap does not measure the effect of changing the evaluation format on the same set of cases. These issues affect the precision of the calibration and interface comparisons, rather than the existence of the underlying effects.

Questions for authors
  1. Which SQ-LLM checkpoint and input wrapper were used, and why was its documented two-audio comparison interface not used? Please clarify whether that interface is supported by the evaluated release and, if so, whether a limited comparison on the same audio pairs changes the reported conclusion.

  2. What does the 30 dB endpoint in Table 11 represent, given the conditions specified in Section 4.3 and the clean endpoint in Figure 3? If the clean condition was assigned a finite value, how do the TES estimates and their uncertainty change when fitting only the actual finite-SNR conditions?

  3. Could you compare pointwise and pairwise accuracy on the same subset of pairs with unequal pointwise scores, alongside an all-pairs comparison with an explicit tie policy? This would clarify both the magnitude of the format effect and the claim that GRM’s pairwise and pointwise results agree most closely.

2 comments
AuthorsOpenReview ↗

To Reviewer dsYt,

We thank the reviewer for the careful reading, including the check against the upstream repository. The points shared with other reviewers are answered in parts 1 and 2 of our response to all reviewers. This reply covers what is specific to your review.

W1 and Q1, the checkpoint and the input wrapper. The checkpoint is the released Hui519/speechllm-as-judge-qwen25omni. Pointwise experiments use its documented single-clip evaluation turn. Pairwise experiments use the concatenated presentation.

Concatenation was a deliberate choice. Four of the five concatenated judges accept a single audio input, so it is their only route to a comparison. We held one presentation fixed across all five so that position effects were comparable, with GRM's native two-input path as the contrast. Table 1 records this in the Input pairing column, and the revision will add the reasoning to the text.

You asked whether a limited comparison on the released interface changes the conclusion. We ran it, and the result is in part 2. On the released interface SQ-LLM reaches 0.989 accuracy and names slot A on 0.511 of trials, against 0.54 and 0.96 concatenated, with every reply decidable under both slot orders. Section 5.4 already assigns SQ-LLM's concatenated failure to the interface, and this result confirms that reading directly. The revision will add it and qualify the abstract's statement about position. One checkpoint moving from 0.54 to 0.989 by packaging alone is direct support for the paper's argument that a judge must be tested on the interface it is used with. As you noted, the acoustic and textual intervention results are unaffected.

The comparison template returns a prose assessment and not an A or B token, so it needs a decision parser. The parser decides every reply, at an identical rate under both slot orders.

W2 and Q2, the 30 dB endpoint. Answered in part 1. Clean speech was placed at 30 dB so that it could enter the regression, and the revision will state this. The refit on the nine measured noise levels leaves the ordering across judges unchanged.

W2 and Q3, the Table 15 denominators. Answered in part 1 with both accountings you asked for. The pointwise advantage holds under both for SQ-LLM, SALMONN-SQA and QualiSpeech. Under the tie policy GRM's gap is -0.08, the only negative value in that column, which strengthens the statement that GRM's pairwise and pointwise results agree most closely.

On the wording controls you credit in Strength 2. We can now report that sweep more completely for SQ-LLM. All four wordings assert the same thing about quality and differ only in how they phrase it, and all four share an identical neutral control. Suggestibility is the positive score minus the negative one.

Caption wordingNegativePositiveSuggestibility
Provenance, used in the paper2.9304.1651.235
Quality2.8404.4801.640
Scenic2.9504.4051.455
Mean opinion score1.9804.4152.435

The neutral score is 3.675 in every row, which is the control holding exactly. The provenance wording used in the paper produces the smallest effect of the four, so the published figures are the conservative choice. This currently covers SQ-LLM only and the revision states that scope.

Dear reviewer dsYt, We are curious whether our additional results and analyses have addressed your concerns regarding our manuscript. To the best of our knowledge, author-review communication ends on 11/18, thus we are keen to learn if there are any additional analysis or supplementations we can make to move the rating to 4. Thank you.

Comments on the paper

5 comments
AuthorsOpenReview ↗

To all reviewers

Response to all reviewers, part 1 of 3: the Table 15 and TES analyses

We appreciate all three reviewers. Three concerns were shared across reviews and we answer them first, followed by a separate reply to each reviewer. Part 1, this one, gives the two analyses that were requested. Part 2 reports new experiments on SQ-LLM's comparison interface. Part 3 lists what changes in the paper as a result, including one claim.

1. Pointwise against pairwise on a common denominator (dsYt Q3, emZJ W1)

Table 15 reports pairwise accuracy over all trials, and pointwise accuracy over the pairs a judge scored unequally, with the tie rates beside them. Both reviewers asked for the two formats on a common denominator. We report that here in both forms they proposed.

Our recomputation matches Reviewer emZJ's figure of 0.654. For SQ-LLM, 48 percent of pairs showed unequal pointwise scores and accuracy on those was 0.82, so giving a tied pair half credit gives 0.48 x 0.82 + 0.52 x 0.50 = 0.65. Against its pairwise accuracy of 0.54, that is an improvement of 0.11 over all pairs, against 0.28 on the decided pairs.

Below, pointwise means that tie-weighted quantity. An unequal pair counts as correct if the higher score went to the clean clip. An equal pair counts as half correct. Every pair is counted, so the denominator matches the pairwise one.

JudgePairwisePointwiseGapAs published
Qwen2-Audio0.500.51+0.01+0.08
Qwen2.5-Omni0.500.50-0.01-0.02
SQ-LLM0.540.65+0.11+0.28
GRM0.640.56-0.08+0.07
SALMONN-SQA0.500.83+0.33+0.48
QualiSpeech0.500.71+0.21+0.39

Reviewer dsYt also asked for both formats on the identical subset. Below, both accuracies are restricted to pairs scored unequally, so no tie policy is needed. Decided is the fraction of pairs in that subset.

JudgeDecidedPairwisePointwiseGap
Qwen2-Audio0.170.500.58+0.08
Qwen2.5-Omni0.210.500.48-0.02
SQ-LLM0.480.570.82+0.24
GRM0.290.680.71+0.02
SALMONN-SQA0.690.500.98+0.48
QualiSpeech0.530.500.89+0.39

The pointwise advantage holds under both accountings for SQ-LLM, SALMONN-SQA and QualiSpeech. Under the tie policy GRM's gap is -0.08, the only negative value in that column, which strengthens our statement that GRM's two readings agree most closely. Table 15 in the revision will carry both accountings and an explicit tie policy.

2. The 30 dB endpoint and the TES fits (dsYt Q2, emZJ W2)

Both reviewers asked what the 30 dB endpoint in Table 11 represents. Clean speech was placed at 30 dB so that it could enter the regression. The revision will state this, and we also refit without it.

By finite conditions we mean the nine measured levels only, -20 to +20 dB in 5 dB steps. Refitting on those drops the clean condition, so no point enters at an assigned coordinate. Values are in decibels except the slope ratio. Matched range is the horizontal separation between the two fitted lines across the scores the judge actually produces.

JudgePublishedFinite onlySlope ratioMatched range
SQ-LLM32.527.81.0631.8 to 32.1
GRM11.410.21.1110.3 to 13.8
SALMONN-SQA2.83.11.46-1.4 to 12.7
QualiSpeech10.77.12.0812.9 to 23.0

The ordering across judges is unchanged. Reviewer emZJ also observed that Equation 4 divides an intercept difference by the average slope even when the slopes differ, so the lines define no single horizontal separation. For QualiSpeech the slopes are 0.0830 and 0.0399 points per decibel, the ratio of 2.08 above.

Where the slopes are close, a single value is well defined. SQ-LLM's matched range spans 0.3 dB. Where they diverge, as for QualiSpeech, we report the range of 12.9 to 23.0 dB in place of a single value, and the comparison between QualiSpeech and GRM is stated in terms of ranges.

The revision will fit finite conditions only and define the headline figure as the separation at a fixed reference score of 3.0, so one value is always recoverable. It will print the matched range and slope ratio beside it. The claim that a textual claim can offset tens of decibels holds for SQ-LLM. For QualiSpeech it is restated as a range.

Response to all reviewers, part 2 of 3: new experiments on the comparison interface

Two reviewers asked about SQ-LLM's comparison interface. Reviewer dsYt noted that the template in SpeechLLM-as-Judges uses two separately supplied audio inputs, and asked which checkpoint and wrapper we used, why concatenation was used instead, and whether a limited comparison on the same pairs changes the conclusion. Reviewer emZJ asked whether SQ-LLM was evaluated through its released comparison format, and whether there is evidence that concatenated-input judges identify both candidate segments, stating that either a supported-interface result or a direct boundary-recognition control would materially change their assessment of the position-bias claim.

We ran both experiments that Reviewer emZJ named. The first evaluates SQ-LLM on its released two-audio comparison interface. The second tests whether judges given a concatenated recording identify both clips. The checkpoint is the released Hui519/speechllm-as-judge-qwen25omni with the ms-swift reasoning wrapper, which supports the two-audio interface. The interface changes the result, and the change supports the paper's argument.

MeasureConcatenatedNative
Accuracy, prefers clean clip0.540.989
Names slot A0.960.511
Decision raten/a1.000

How the slot figure is computed, and what to compare it against

Each pair is presented twice, once with the clean clip in slot A and once in slot B. Names slot A is the fraction of decidable trials naming slot A, pooled over both orders. Because the clean clip occupies slot A on exactly half the trials by construction, a judge responding to the audio gives 0.5 and a judge responding to position approaches 1 or 0. So 0.5 is the reference point, not a null hypothesis under test.

The pooled figure alone is insufficient, since a judge that always answers correctly and one that answers at random both give 0.5. What matters is whether it moves with the clean clip.

ConditionDecision rateAccuracyNames slot A
Clean clip in slot A1.0001.0001.000
Clean clip in slot B1.0000.9780.022

The slot named tracks the clean clip almost perfectly. A position-driven judge would name slot A at a high rate under both orders, which is what concatenation produces. In Table 7 SQ-LLM picks the first slot in both orders on 0.919 of pairs, which is 0.96 of trials. The decision rate is identical across orders, so this is also not an artifact of how prose replies are read into a decision. This is a 30-clip pilot, 180 judgments. The 200-clip replication on the paper's full clip set is under way and we will report it in this thread. We do not expect the direction to change, since the effect is the difference between 0.54 and 0.989 against a chance baseline of 0.5 and the decision rate was 1.000 under both orders.

The judges are not failing to hear the two clips

Reviewer emZJ's second question and Reviewer yK9S's W3 both ask what the failure actually is. Four arms on identical pairs, so every comparison is internal to one run: the published prompt at a 0.5 s separator, the same at 2.0 s, a question asking which half carries the added noise, and a question asking how many clips are heard. The last two have verifiable answers, which is what makes them diagnostic. Says two is the fraction reporting exactly two clips; Locate is accuracy at naming the degraded half.

JudgeSays two clipsLocate accuracySlot A on locateSlot A on preference
SQ-LLM0.9670.9110.4670.872
Qwen2.5-Omni1.0000.6280.8611.000
Qwen2-Audio1.0000.5000.0000.000
SALMONN-SQA0.4560.5001.0001.000
QualiSpeech1.0000.5001.0001.000

Raising the separator to 2.0 s moved SQ-LLM's preference figure only to 0.844, so separator length was not the obstacle.

Three distinct failures appear, which is the distinction Reviewer yK9S asked us to keep. SQ-LLM reports two clips and names the degraded half on 0.911 of trials with a near-balanced slot figure, then answers the preference question from position on 0.872, so its perception of the two clips is intact and does not reach its comparative answer. SALMONN-SQA reports three clips on 98 of 180 trials, so there the presentation genuinely prevents segmentation. Qwen2-Audio counts correctly but emits one letter on every comparison, so its 0.500 is a single repeated response rather than discrimination.

This control uses noise at three severities on 30 clips. Its baseline arm is therefore not numerically comparable to the published Table 15, which covers five degradation types on 200 clips. The comparisons in the table above are all within this one control.

What this changes in the paper is in part 3.

Response to all reviewers, part 3 of 3: what changes in the paper

The pairwise claim

Four of the five concatenated judges are single-clip models with no two-input interface of any kind. For them concatenation is not one option among several, it is the only route to a pairwise judgement, so the published result remains the only pairwise result they can produce and none of their numbers change. SQ-LLM is the one judge with a documented comparison template, and the one number that moves.

Section 5.4 already assigns SQ-LLM's concatenated failure to the interface. We will add the native-interface result there as direct evidence, and qualify the abstract's statement that five of six judges rely on position. We thank Reviewers dsYt and emZJ for the question that led to this result.

The result strengthens the paper's argument in three ways.

Pairwise competence tracks the availability of a native two-input path. The two judges that have one, GRM at 0.64 and SQ-LLM at 0.989, are the only two exceeding chance.

The size of that dependence is itself the result. One checkpoint yields 0.54 or 0.989 according only to how two clips are packaged. For a model used to score data or supply a training reward, the harness an engineer happens to build therefore sets the quality of their evaluation, which is a more consequential warning than position bias in one model.

And it is not a perception limit, because the judge localises the degradation in the very presentation where its preference is position-driven.

This argues our thesis more directly than the original framing did. A judge has to be evaluated on the interface and the decision it will actually be used for.

Other revisions

The revised paper will make the following changes. Table 15 is rebuilt with both accountings and an explicit tie policy, and the sentence about GRM's two readings is restated in terms of the tie-weighted gap. The TES fits use finite conditions only, state the clean handling explicitly, and report a matched range and slope ratio beside every value, with the headline figure defined at a fixed reference score of 3.0. Appendix I is promoted into the main text, since it separates two failure mechanisms. The Related Work passage on Park et al. is rewritten to describe its interventions accurately. The system-level recommendation is narrowed to what Table 4 and the objective-metric comparison support, which is that per-condition averaging reduces per-clip noise on the degradations we constructed. Ranking real synthesis or enhancement systems is stated as outside the scope of this probe. All citations are corrected, and the long paragraphs in Sections 5 and 6 are broken up.

One experiment still running

We will report it in this thread. It is a free-text probe on the silence conditions. Reviewer yK9S observed that a high score for silence does not establish that a judge cannot recognise silence, since it may treat the absence of noise as a positive feature or may not know it should decline to answer. Experiment B instructs the judge to answer with only a number, so its replies carry no evidence about what it noticed, and no reanalysis of the submitted data can separate those accounts. The probe reruns the same three conditions, which are real speech, digital silence, and white noise matched in level to the speech with no speech present, with the instruction relaxed so the judge describes the recording before rating it. Each description is coded for whether the judge states an absence of speech, describes speech that is not there, mentions noise, or declines to rate, with the raw replies retained so any claim can be checked against the text. A judge that states the recording is empty and still rates it above 3 is rewarding the wrong thing. A judge that describes speech is not perceiving the condition. A judge that declines knows the question does not apply. Those call for different remedies, which is the distinction the reviewer asked us to keep.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.