- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper examines whether text-output scores are sufficient to evaluate quantized speech models on tasks where the target label cannot be determined from the transcript. The authors evaluate fixed 6-bit and 7-bit mixed-precision allocations of Qwen2-Audio-7B-Instruct, selected using a separate 387-example speech-translation development set, on 508 English-to-German FLEURS utterances and 512 RAVDESS emotion clips from 16 speakers. The RAVDESS subset balances two spoken sentences across eight emotion labels, making sentence identity insufficient to determine emotion. The BLEU and chrF differences from FP16 remain statistically unresolved at both bit budgets. However, the selected 6-bit allocation reduces emotion accuracy from 70.12% to 66.41%, with a speaker-level confidence interval excluding zero and an exact sign-flip p-value of 0.0148. The 7-bit FP16 comparison remains unresolved. Both matched-budget controls achieve higher emotion-accuracy point estimates than the selected allocations at their respective budgets. The authors argue that text-based scores alone cannot establish performance on lexically non-diagnostic speech tasks.
Critical strengths
-
Careful statistical analysis and transparent reporting. The authors appropriately estimate uncertainty over the 16 speakers rather than treating all 512 clips as independent. The reported accuracies are consistent with the per-speaker results in Table 3, allowing for rounding, and the exact sign-flip calculations are reproducible. The paper also documents its scoring procedures and output provenance and avoids interpreting confidence intervals containing zero as evidence of equivalence. These choices make the main finding credible within the evaluated setting.
-
Meaningful matched-budget controls. The front-block and stratified-uniform controls examine precision placement at nearly identical average bit budgets. The authors transparently report that the selected allocations have lower emotion-accuracy point estimates than both controls at both budgets. This provides useful evidence that average bit width alone is insufficient to characterize quantization performance.
Critical weaknesses
-
Limited novelty and scientific contribution. The distinction between lexical and acoustic information has already been examined in prior work, including LISTEN (Chen et al., EACL 2026), which uses controlled cue-conflict experiments. The authors acknowledge this work and recognize that their balanced-sentence design cannot establish acoustic cue reliance. Their main contribution therefore rests on the quantization case study. Although the 6-bit deficit is statistically supported, it represents a net difference of 19 correct predictions out of 512, evaluated on one checkpoint, one acted-emotion corpus, one prompt, and 16 speakers. The study neither identifies the mechanism behind the degradation nor establishes whether similar behavior occurs across other models or non-lexical tasks. The finding is useful, but the additional scientific insight appears limited. The presentation compounds this: the same interpretive caveats are restated across Sections 3.1, 4.4, 4.5, 5 and 7, the conclusion extends over a page of repeated protocol advice, and the module-selection criterion central to the method cannot be recovered from the text at all, which is the basis for my presentation score.
-
Incomplete cross-task validation and error analysis. FLEURS results are reported only for FP16 and the two selected allocations. Translation results for the four matched-budget controls are missing. Consequently, the study does not establish whether the translation-selected allocations actually offer an advantage over alternative precision placements on translation, even though those alternatives achieve better emotion-accuracy point estimates. This is important because the claimed cross-task distinction would be more convincing if the translation-based selection procedure were shown to provide a measurable translation benefit relative to the same-budget controls. Furthermore, the 7-bit front-block control reaches 73.44% emotion accuracy compared with 70.12% for FP16, yet paired FP16 comparisons for the controls are not reported. Finally, the canonical single-label scoring rule may count invalid responses as errors, but the paper provides neither invalid-output rates nor confusion matrices. It remains unclear whether the observed degradation primarily reflects emotion misclassification or response-format changes. These gaps substantially limit the interpretation of the empirical contribution.
Questions for authors
- Could the authors report BLEU and chrF for all four matched-budget controls on the same 508 FLEURS utterances? Do the translation-selected allocations outperform these alternatives on translation?
- Could the authors provide paired RAVDESS comparisons between FP16 and each control, including confidence intervals? In particular, how should the 73.44% accuracy of the 7-bit front-block control be interpreted relative to FP16?
- How many outputs from each allocation fail the canonical single-label scoring rule? Could the authors provide class-level confusion matrices to distinguish emotion misclassification from response-format errors?
- What specific objective or selection criterion determined which modules were upgraded to 8-bit precision using the 387-example translation development split?
- What motivated the restriction to actors 1–16 rather than all 24 RAVDESS actors? Was this subset fixed before evaluation, and how might the selection affect the scope of the conclusions?
4 comments
Thank you for the detailed response and clarifications. I appreciate the authors' careful reporting and acknowledge that the speaker-level analysis supports the 6-bit emotion-accuracy decline. However, four concerns remain.
First, the missing FLEURS results for matched-budget controls leave the translation advantage of the selected allocations unclear. Second, paired FP16 confidence intervals are unavailable for the controls, particularly the 7-bit front-block allocation (73.44% versus 70.12%). Third, the absence of invalid-output rates and confusion matrices makes it difficult to distinguish classification errors from response-format errors. Fourth, the rationale for selecting actors 1–16 and the criterion for assigning 8-bit precision to modules remain unspecified.
While these limitations do not invalidate the main finding, they restrict its interpretation and broader scientific contribution. I therefore maintain my rating.
Thank you. The four points you identify remain unresolved: the paper reports no FLEURS matched-control BLEU/chrF results, no paired FP16 intervals for the controls, no invalid-output rates or confusion matrices, and no rationale for the selected actor subset or module-level bit assignments. Our previous response acknowledged parts of this gap but did not provide the requested evidence or a manuscript revision. We appreciate your assessment and understand that you maintain your rating.
Thank you for specifying the remaining gaps. The revised PDF and source supplement are attached. They now report FLEURS controls, the RAVDESS control comparisons, output validity, confusion counts, and the allocation procedure.
On the same 508 FLEURS test utterances, BLEU/chrF are 20.327/49.680 for both 6-bit controls and 20.336/49.659 for both 7-bit controls. Against the selected allocations, the 6-bit controls are +0.111 BLEU [−0.001,+0.271] and +0.079 chrF [+0.004,+0.179]; the 7-bit controls are +0.009 BLEU [−0.140,+0.132] and −0.021 chrF [−0.108,+0.053]. These results show no consistent held-out translation advantage for the selected maps.
For RAVDESS, we added a post hoc 24-actor analysis (768 clips per arm). It includes the primary 16 actors, so it is an overlapping sensitivity analysis, not an independent replication. Against FP16 at 69.79%, paired actor-bootstrap accuracy differences are: front-6 67.97%, −1.82% [−3.13,−0.65]; uniform-6 68.49%, −1.30% [−2.60,−0.26]; front-7 72.79%, +2.99% [+1.82,+4.04]; uniform-7 69.92%, +0.13% [−0.78,+1.04]. Thus the primary-set 73.44% front-7 estimate is lower on the full set, while its full-set interval is above zero; the extension remains post hoc. All seven arms returned canonical labels for 768/768 items (zero invalid outputs); the revision includes per-emotion recall and full confusion counts.
The allocation objective is maximum mean chrF across seven specified plans on the separate 387-example development set. The appendix gives exact decoder-layer masks and module counts; quantization is symmetric per-channel round-to-nearest with dequantized inference. The available project record does not establish why actors 1–16 were initially selected or whether that subset was fixed before outcome review. We state this directly and limit interpretation of the added 24-actor analysis accordingly.
The paper's contribution is a quantization case study of a cross-task inference. It cites LISTEN for lexical-versus-acoustic cue-conflict evaluation; this paper tests whether stable translation output scores establish preserved emotion-label performance for the same selected allocations. One Qwen2-Audio checkpoint and one acted-emotion subset do not establish how often this pattern occurs across models or tasks.
The four matched-budget controls are evaluated on RAVDESS, not FLEURS. Relative to FP16 at 70.12%, their point differences are −1.96% (68.16% − 70.12%) and −1.76% (68.36% − 70.12%) at 6 bits, and +3.32% (73.44% − 70.12%) and +0.19% (70.31% − 70.12%) at 7 bits. Appendix Table 4 reports selected-versus-control intervals, not paired control-versus-FP16 intervals; uncertainty for those differences is unavailable. Selected-allocation accuracies are 66.41% and 68.95%. The selected 7-bit versus FP16 result is −1.17% [−2.73%, +0.20%], p=0.2656 in Section 4.2 and Figure 2, but that comparison is omitted from Appendix Table 4. FLEURS results for controls are absent, so the study does not show translation-based selection beating same-budget controls on translation.
The paper specifies actors 1–16 and four clips per actor-emotion pair, but not the selection rationale or intensity/repetition subset. It gives the canonical scoring rule but no invalid-output rates or confusion matrices; raw predictions are unavailable for this check. Table 3 contains five negative, nine zero, and two positive 7-bit actor-level differences.