- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether audio-language models used as speech-quality judges respond to acoustic evidence, textual quality claims, or presentation order. It evaluates six released models using controlled interventions derived from 200 LibriSpeech utterances, with selected experiments repeated on 195 VCTK utterances. The experiments cover graded acoustic degradation, replacement with silence or noise, conflicting textual claims, and order-swapped pairwise comparisons. The paper reports that quality fine-tuning improves degradation sensitivity but does not consistently eliminate prompt sensitivity, inappropriate scores for silence, or failures under concatenated comparison. It also introduces a text-equivalent SNR statistic intended to express textual influence in acoustic units.
Critical strengths
-
The matched interventions expose failures that aggregate agreement can miss. The same utterances are evaluated across acoustic and textual conditions, reducing confounding from speaker and recording differences. Tables 3 and 6 show that a model can respond strongly to degradation while remaining susceptible to a quality claim: SQ-LLM has a white-noise ASI of 0.77, yet its clean-audio score changes by 1.23 points between the two claims. This is a useful distinction for selecting and auditing speech evaluators.
-
The paper checks several plausible explanations for its main findings. Recorded noise supplements synthetic white noise, selected effects reproduce on VCTK, and Appendix F tests alternative claim wordings. Appendix C also resamples utterances together with their derived conditions rather than treating all scores as independent observations. These choices strengthen the evidence that the reported acoustic and textual effects are not confined to one wording or one corpus.
Critical weaknesses
-
The pairwise experiment does not adequately distinguish positional preference from failure to interpret the comparison format. Five judges receive one concatenated waveform with a 0.5-second silent separator. Appendix H explicitly recognizes that a judge may fail to locate the boundary, but scoring the clips separately does not resolve that possibility. It establishes useful single-clip discrimination, not successful identification of both candidates during comparison. In section 5.4, the authors appropriately attributes some failures to the interface, but the stronger interpretation as position bias remains underdetermined. This matters particularly for SQ-LLM. The pointwise comparison also uses a different denominator. In Table 15, SQ-LLM’s 0.82 accuracy applies only to the 48% of pairs with unequal scores, whereas its 0.54 pairwise accuracy covers all trials. Assigning half credit to pointwise ties gives approximately , so the comparable improvement is about 0.114 rather than the displayed 0.28. The improvement survives, but the reported gap overstates the practical recovery.
-
TES is not sufficiently justified as an equivalent acoustic shift. Section 4.3 specifies noisy conditions from −20 to 20 dB plus clean speech, yet Table 11 reports fitting ranges extending to 30 dB and a total span of 50 dB. Figure 3 correctly notes that clean speech has no finite SNR. The manuscript therefore appears to assign the clean condition an artificial finite coordinate without explaining or testing that choice, which can change the fitted slopes and TES. Additionally, Eq. 4 divides an intercept difference by the average slope even when the two slopes differ substantially. For QualiSpeech, Table 11 reports 0.0830 and 0.0399 points/dB. Such lines do not define a single horizontal separation: the equivalent shift depends on the score being matched. The paper acknowledges that TES is approximate, but does not quantify the approximation error or demonstrate matched-score support for the reported shifts. This weakens the interpretation of values such as 32.5 dB as degradation that a textual claim can compensate for.
Questions for authors
-
Was SQ-LLM evaluated using its released comparison format, and is there evidence that the concatenated-input judges identify both candidate segments? A supported-interface result or a direct boundary-recognition control would materially change my assessment of the position-bias claim. Please also compare pointwise and pairwise performance on a common denominator, with an explicit tie policy.
-
Please clarify the coordinate assignment and whether the reported values remain stable when clean speech is excluded. For unequal slopes, what matched-score range supports the claimed equivalent shifts without extrapolation? Stability under these checks would substantially strengthen the proposed metric.
4 comments
Thank you for the additional experiments. The native-interface and boundary-recognition results substantially address my first concern, and the common-denominator analysis resolves the comparison issue. My remaining concern is regarding TES: fixing the reference score at 3.0 does not itself guarantee a comparison without extrapolation. Please report the SNR implied by each fitted line at this score, verify that both lie within the retained finite-SNR support, and clarify how the finite-only TES values relate to the matched ranges. I am retaining my current rating for now, but would consider move to 3 if this support is established or the corresponding claims are appropriately narrowed.
Thank you, reviewer emZJ, for your response and time for checking our comments. We are grateful that you are considering the move to 3. We promise that we will try our absolute best to meet your additional comments. We are beginning to prepare additional analysis. Let us come back with the results and either 1) supplement the results or 2) narrow our claims, depending on the results, which we will share either way.
We would like to share one information, which is purely about timing. As far as we are concerned, "Author-Reviewer Discussion Period" ends 11/18 (tomorrow). While we are starting this additional analysis right away, we might not be able to come back within the next 24 hours. We will comeback with results and either supplement or narrow our claim on TES. In the meantime, we would greatly appreciate if reviewer could consider moving the score to 3 given the current circumstances (other concerns have been resolved and our claims regarding TES could be either supplemented or narrowed based on additional analysis). Thank you.
Thank you for the follow-up. You are right about the reference score, and the check below confirms it. We also think the check sharpens the paper's claim rather than weakening it and we explain below
The SNR implied at 3.0. Values are in decibels. GRM uses a 1 to 10 scale.
| Judge | Retained SNR support | Studio line at 3.0 | Phone line at 3.0 | Both inside support |
|---|---|---|---|---|
| SQ-LLM | -5 to 20 | -0.2 | 29.0 | No |
| GRM | -10 to 20 | -9.9 | -1.4 | Yes, at the edge |
| SALMONN-SQA | -15 to 20 | 9.1 | 20.2 | Marginal |
| QualiSpeech | 0 to 20 | 8.6 | 32.9 | No |
Only GRM has both values inside the support, and its studio value sits at the lower edge. A fixed score of 3.0 therefore does not avoid extrapolation, and we drop it.
Finite-only values and matched ranges. Our earlier table mixed two fits, and we apologise. The matched ranges came from the published fits, which include the clean point at 30 dB. The finite-only column came from refits without it. On the finite-only fits, restricted to scores both lines reach inside the support, the ranges are 8.4 to 14.2 for GRM, 12.1 to 16.7 for QualiSpeech and -4.0 to 11.1 for SALMONN-SQA. The Eq. 4 value equals the horizontal separation only for parallel lines, so it need not fall inside these ranges.
Why SQ-LLM has no in-support value. No score is reached by both SQ-LLM lines inside the support. Studio-labelled audio at -5 dB scores 2.69 on average. Phone-labelled audio at +20 dB scores 2.58. The label effect spans the whole acoustic range we measured. Extrapolation was needed only to say how far beyond it the effect reaches. The failed check reflects the limits of our SNR range. It is not evidence against the effect.
The claim does not depend on the conversion. TES converts a measured score gap into decibels. The paper's argument rests on the gap itself, which is the text effect set against the acoustic effect on the same clips. That comparison needs no fitted line. The table divides the mean score gap between the studio and phone labels by the score change across the retained SNR range. Brackets are bootstrap intervals over the 200 clips.
| Judge | Label gap | Acoustic change under the labels | Ratio | Ratio against neutral |
|---|---|---|---|---|
| SQ-LLM | 1.46 | 1.35 | 1.09 [1.00, 1.19] | 0.64 [0.61, 0.68] |
| QualiSpeech | 1.13 | 2.08 | 0.55 [0.51, 0.58] | 0.48 [0.45, 0.51] |
| GRM | 0.93 | 2.75 | 0.34 [0.30, 0.38] | 0.43 [0.37, 0.50] |
| SALMONN-SQA | 0.25 | 2.48 | 0.10 [0.09, 0.11] | 0.10 [0.09, 0.11] |
The fourth column uses the larger of the two label curves' changes, which is the conservative choice. The last column uses the change under the neutral prompt instead. For SQ-LLM the label moves the score as far as the full 25 dB of added noise does. The two labels also shrink SQ-LLM's response to noise, from 2.28 points under the neutral prompt to 1.35 and 1.25. So the text weakens the judge's response to the audio as well as shifting the score. SQ-LLM is the largest and SALMONN-SQA the smallest under both columns. QualiSpeech and GRM overlap under the neutral column, and the revision does not order them.
Decibel figures that stay in support. Against the neutral prompt, the studio label alone is worth about 15 dB for SQ-LLM, with a bootstrap interval of 13.9 to 16.1. This comes from the three lowest retained levels, where a match exists. SQ-LLM also scores clean speech labelled as a phone recording like studio-labelled speech at -1.7 dB. That figure uses the clean condition without assigning it a coordinate.
The revision. The headline measure becomes the ratio above, which needs no extrapolation. Decibel figures are reported only where both curves share a score inside the measured range, with the matched range and slope ratio beside them. For SQ-LLM the paper states that the label effect spans the full 25 dB range we tested. It reports 27.8 as an extrapolation beyond that range. The conclusion stands. A text claim moves SQ-LLM's score as far as 25 dB of real noise does. All numbers come from scripts/tes_support_check.py in the released code. We hope this narrows the TES claims in the way you described.
To Reviewer emZJ,
We thank the reviewer for the precise technical comments. You wrote that a supported-interface result or a boundary-recognition control would materially change your assessment of the position-bias claim. We ran both, along with the two analyses you asked for. The detail is in parts 1 to 3 of our response to all reviewers.
W1, position preference against failure to interpret the format. Section 5.4 already separates two readings of the concatenated result. It assigns SQ-LLM, SALMONN-SQA and QualiSpeech to the reading in which the judge can order the recordings and the interface loses that ordering. Table 7 supports this. Every pair is judged in both orders, and SQ-LLM picks the first slot both times on 0.919 of pairs, with a position-immune discrimination of 0.08, so its answer follows the slot and not the recording. You asked for direct evidence, and the two new controls provide it.
On its released two-audio interface SQ-LLM reaches 0.989 accuracy and names slot A on 0.511 of trials, against 0.54 and 0.96 concatenated, with every reply decidable and an identical decision rate under both slot orders. In the identification control SQ-LLM reports two clips on 0.967 of trials and names the degraded half on 0.911 with a near-balanced slot figure, yet answers the preference question from position on 0.872. Its perception of the two clips is intact and does not reach its comparative answer. Raising the separator from 0.5 to 2.0 seconds moved that figure only to 0.844, so separator length was not the obstacle. SALMONN-SQA reports three clips on 98 of 180 trials, so for that judge the presentation does prevent segmentation. Both tables are in part 2.
The revision will add both results to Section 5.4, keep the two causes apart, and qualify the abstract's statement that five of six judges rely on position. The numbers of the other four concatenated judges do not change, since they accept a single audio input.
On the denominator, our recomputation matches your figure of 0.654, which gives an improvement of 0.11 over all pairs. As you noted, the improvement remains. The full accounting is in part 1.
W2, the justification for TES. The paper presents TES as an approximation, and your two checks let us quantify it.
On the coordinate, clean speech was placed at 30 dB so that it could enter the regression, and the revision will state this. The refit on the nine measured noise levels is in part 1 and leaves the ordering of the judges unchanged.
On unequal slopes, the slope ratios and matched-score ranges are in part 1. Where the slopes are close a single value is well defined, and SQ-LLM's matched range spans 0.3 dB, from 31.8 to 32.1. Where they diverge, as for QualiSpeech with a ratio of 2.08, we report the range of 12.9 to 23.0 dB in place of a single value.
The revision will define the headline figure as the horizontal separation at a fixed reference score of 3.0, so one value is always recoverable without extrapolation, and will report the matched range and slope ratio beside it. The statement that a textual claim can offset tens of decibels holds for SQ-LLM. You wrote that stability under these checks would substantially strengthen the metric, and the ordering of the judges is stable under both.