acceptodds
Under review as a conference paper at ICLR 2027

Text Scores Do Not Establish Performance on Lexically Non-Diagnostic Speech Tasks: A Qwen2-Audio Quantization Case Study

Abstract

Text-output scores alone do not show whether quantization preserves performance when the transcript does not reveal the target label. We evaluate fixed 6- and 7-bit Qwen2-Audio-7B-Instruct allocations on 508 English-to-German FLEURS utterances and on 512 RAVDESS emotion clips from 16 speakers. The provenance-bound FLEURS rerun has selected-minus-FP16 BLEU and chrF intervals that all include zero. On RAVDESS, where the same two sentences occur equally often with every emotion label, canonical-label accuracy is -3.71% for 6 bit and -1.17% for 7 bit relative to FP16. The 6-bit speaker interval excludes zero and an exact two-sided sign-flip test gives p=0.0148; the 7-bit interval includes zero. Same-budget controls do not identify either selected allocation as best. This case study shows that a translation score does not establish performance on a speech task whose target cannot be recovered from the transcript.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
speech-language models, quantization, speech translation, emotion recognition, audio evaluation, cross-task evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_4bG6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines whether text-output scores are sufficient to evaluate quantized speech models on tasks where the target label cannot be determined from the transcript. The authors evaluate fixed 6-bit and 7-bit mixed-precision allocations of Qwen2-Audio-7B-Instruct, selected using a separate 387-example speech-translation development set, on 508 English-to-German FLEURS utterances and 512 RAVDESS emotion clips from 16 speakers. The RAVDESS subset balances two spoken sentences across eight emotion labels, making sentence identity insufficient to determine emotion. The BLEU and chrF differences from FP16 remain statistically unresolved at both bit budgets. However, the selected 6-bit allocation reduces emotion accuracy from 70.12% to 66.41%, with a speaker-level confidence interval excluding zero and an exact sign-flip p-value of 0.0148. The 7-bit FP16 comparison remains unresolved. Both matched-budget controls achieve higher emotion-accuracy point estimates than the selected allocations at their respective budgets. The authors argue that text-based scores alone cannot establish performance on lexically non-diagnostic speech tasks.

Critical strengths
  • Careful statistical analysis and transparent reporting. The authors appropriately estimate uncertainty over the 16 speakers rather than treating all 512 clips as independent. The reported accuracies are consistent with the per-speaker results in Table 3, allowing for rounding, and the exact sign-flip calculations are reproducible. The paper also documents its scoring procedures and output provenance and avoids interpreting confidence intervals containing zero as evidence of equivalence. These choices make the main finding credible within the evaluated setting.

  • Meaningful matched-budget controls. The front-block and stratified-uniform controls examine precision placement at nearly identical average bit budgets. The authors transparently report that the selected allocations have lower emotion-accuracy point estimates than both controls at both budgets. This provides useful evidence that average bit width alone is insufficient to characterize quantization performance.

Critical weaknesses
  • Limited novelty and scientific contribution. The distinction between lexical and acoustic information has already been examined in prior work, including LISTEN (Chen et al., EACL 2026), which uses controlled cue-conflict experiments. The authors acknowledge this work and recognize that their balanced-sentence design cannot establish acoustic cue reliance. Their main contribution therefore rests on the quantization case study. Although the 6-bit deficit is statistically supported, it represents a net difference of 19 correct predictions out of 512, evaluated on one checkpoint, one acted-emotion corpus, one prompt, and 16 speakers. The study neither identifies the mechanism behind the degradation nor establishes whether similar behavior occurs across other models or non-lexical tasks. The finding is useful, but the additional scientific insight appears limited. The presentation compounds this: the same interpretive caveats are restated across Sections 3.1, 4.4, 4.5, 5 and 7, the conclusion extends over a page of repeated protocol advice, and the module-selection criterion central to the method cannot be recovered from the text at all, which is the basis for my presentation score.

  • Incomplete cross-task validation and error analysis. FLEURS results are reported only for FP16 and the two selected allocations. Translation results for the four matched-budget controls are missing. Consequently, the study does not establish whether the translation-selected allocations actually offer an advantage over alternative precision placements on translation, even though those alternatives achieve better emotion-accuracy point estimates. This is important because the claimed cross-task distinction would be more convincing if the translation-based selection procedure were shown to provide a measurable translation benefit relative to the same-budget controls. Furthermore, the 7-bit front-block control reaches 73.44% emotion accuracy compared with 70.12% for FP16, yet paired FP16 comparisons for the controls are not reported. Finally, the canonical single-label scoring rule may count invalid responses as errors, but the paper provides neither invalid-output rates nor confusion matrices. It remains unclear whether the observed degradation primarily reflects emotion misclassification or response-format changes. These gaps substantially limit the interpretation of the empirical contribution.

Questions for authors
  1. Could the authors report BLEU and chrF for all four matched-budget controls on the same 508 FLEURS utterances? Do the translation-selected allocations outperform these alternatives on translation?
  2. Could the authors provide paired RAVDESS comparisons between FP16 and each control, including confidence intervals? In particular, how should the 73.44% accuracy of the 7-bit front-block control be interpreted relative to FP16?
  3. How many outputs from each allocation fail the canonical single-label scoring rule? Could the authors provide class-level confusion matrices to distinguish emotion misclassification from response-format errors?
  4. What specific objective or selection criterion determined which modules were upgraded to 8-bit precision using the 387-example translation development split?
  5. What motivated the restriction to actors 1–16 rather than all 24 RAVDESS actors? Was this subset fixed before evaluation, and how might the selection affect the scope of the conclusions?
4 comments

The paper's contribution is a quantization case study of a cross-task inference. It cites LISTEN for lexical-versus-acoustic cue-conflict evaluation; this paper tests whether stable translation output scores establish preserved emotion-label performance for the same selected allocations. One Qwen2-Audio checkpoint and one acted-emotion subset do not establish how often this pattern occurs across models or tasks.

The four matched-budget controls are evaluated on RAVDESS, not FLEURS. Relative to FP16 at 70.12%, their point differences are −1.96% (68.16% − 70.12%) and −1.76% (68.36% − 70.12%) at 6 bits, and +3.32% (73.44% − 70.12%) and +0.19% (70.31% − 70.12%) at 7 bits. Appendix Table 4 reports selected-versus-control intervals, not paired control-versus-FP16 intervals; uncertainty for those differences is unavailable. Selected-allocation accuracies are 66.41% and 68.95%. The selected 7-bit versus FP16 result is −1.17% [−2.73%, +0.20%], p=0.2656 in Section 4.2 and Figure 2, but that comparison is omitted from Appendix Table 4. FLEURS results for controls are absent, so the study does not show translation-based selection beating same-budget controls on translation.

The paper specifies actors 1–16 and four clips per actor-emotion pair, but not the selection rationale or intensity/repetition subset. It gives the canonical scoring rule but no invalid-output rates or confusion matrices; raw predictions are unavailable for this check. Table 3 contains five negative, nine zero, and two positive 7-bit actor-level differences.

Thank you for the detailed response and clarifications. I appreciate the authors' careful reporting and acknowledge that the speaker-level analysis supports the 6-bit emotion-accuracy decline. However, four concerns remain.

First, the missing FLEURS results for matched-budget controls leave the translation advantage of the selected allocations unclear. Second, paired FP16 confidence intervals are unavailable for the controls, particularly the 7-bit front-block allocation (73.44% versus 70.12%). Third, the absence of invalid-output rates and confusion matrices makes it difficult to distinguish classification errors from response-format errors. Fourth, the rationale for selecting actors 1–16 and the criterion for assigning 8-bit precision to modules remain unspecified.

While these limitations do not invalidate the main finding, they restrict its interpretation and broader scientific contribution. I therefore maintain my rating.

Thank you. The four points you identify remain unresolved: the paper reports no FLEURS matched-control BLEU/chrF results, no paired FP16 intervals for the controls, no invalid-output rates or confusion matrices, and no rationale for the selected actor subset or module-level bit assignments. Our previous response acknowledged parts of this gap but did not provide the requested evidence or a manuscript revision. We appreciate your assessment and understand that you maintain your rating.

Thank you for specifying the remaining gaps. The revised PDF and source supplement are attached. They now report FLEURS controls, the RAVDESS control comparisons, output validity, confusion counts, and the allocation procedure.

On the same 508 FLEURS test utterances, BLEU/chrF are 20.327/49.680 for both 6-bit controls and 20.336/49.659 for both 7-bit controls. Against the selected allocations, the 6-bit controls are +0.111 BLEU [−0.001,+0.271] and +0.079 chrF [+0.004,+0.179]; the 7-bit controls are +0.009 BLEU [−0.140,+0.132] and −0.021 chrF [−0.108,+0.053]. These results show no consistent held-out translation advantage for the selected maps.

For RAVDESS, we added a post hoc 24-actor analysis (768 clips per arm). It includes the primary 16 actors, so it is an overlapping sensitivity analysis, not an independent replication. Against FP16 at 69.79%, paired actor-bootstrap accuracy differences are: front-6 67.97%, −1.82% [−3.13,−0.65]; uniform-6 68.49%, −1.30% [−2.60,−0.26]; front-7 72.79%, +2.99% [+1.82,+4.04]; uniform-7 69.92%, +0.13% [−0.78,+1.04]. Thus the primary-set 73.44% front-7 estimate is lower on the full set, while its full-set interval is above zero; the extension remains post hoc. All seven arms returned canonical labels for 768/768 items (zero invalid outputs); the revision includes per-emotion recall and full confusion counts.

The allocation objective is maximum mean chrF across seven specified plans on the separate 387-example development set. The appendix gives exact decoder-layer masks and module counts; quantization is symmetric per-channel round-to-nearest with dequantized inference. The available project record does not establish why actors 1–16 were initially selected or whether that subset was fixed before outcome review. We state this directly and limit interpretation of the added 24-actor analysis accordingly.

Reviewer_r1HcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates a methodological question in the evaluation of post-training quantized (PTQ) audio-language models: whether preserving text-output quality scores on a lexical speech task (speech translation BLEU/chrF) suffices to establish preserved performance on a lexically non-diagnostic speech task—one where identical transcripts share the same empirical label distribution so the transcript alone carries zero information above chance (1/C1/C). Using Qwen2-Audio-7B-Instruct under simulated {4-bit,8-bit}\{4\text{-bit}, 8\text{-bit}\} module-level round-to-nearest (RTN) weight-only PTQ at nominal 6-bit and 7-bit parameter-weighted budgets, the authors compare two fixed translation-selected bit allocations against FP16 and two deterministic same-budget heuristic controls (Front-block and Stratified-uniform) across 508 FLEURS EN$\to$DE speech translation utterances and 512 balanced-sentence RAVDESS emotion recognition clips from 16 speakers. While both 6-bit and 7-bit translation-selected allocations preserve FLEURS BLEU and chrF within ≤0.15\le 0.15 points of FP16 (with 95% bootstrap CIs spanning zero), the 6-bit translation-selected allocation drops by −3.71%-3.71\% on RAVDESS (66.41%66.41\% vs. 70.12%70.12\%, speaker-cluster 95% CI [−6.05,−1.37]%[-6.05, -1.37]\%, exact sign-flip p=0.0148p = 0.0148), and neither translation-selected plan outperforms the heuristic controls on RAVDESS (with 7-bit Front-block outperforming 7-bit Selected by +4.49%+4.49\%, p=0.0020p = 0.0020).

Critical strengths
  1. Clean Experimental Isolation of Lexical Non-Diagnosticity and Rigorous Speaker-Cluster Inference: By constructing a balanced RAVDESS evaluation where the exact same two lexically neutral sentences appear 32 times in each of the 8 emotion classes across every speaker, the study guarantees mathematically that transcript-only shortcuts cannot exceed the 1/8=12.5%1/8 = 12.5\% chance ceiling. Furthermore, treating the N=16N = 16 speakers (32 clips/speaker) as the primary resampling and permutation unit—via both 10,000-draw speaker-cluster bootstrap CIs and exact enumeration of all 216=65,5362^{16} = 65{,}536 paired actor sign flips—properly accounts for within-speaker correlation that standard clip-level binomial tests ignore.
  2. Disciplined Epistemic Scope: The manuscript carefully distinguishes between simulated (dequantized) weight-only PTQ and packed-weight hardware execution, avoids invalid cross-endpoint difference-in-differences claims between BLEU and accuracy, and explicitly pairs claims with non-claims in Table 1.
Critical weaknesses
  1. Asymmetric Experimental Matrix (Missing Heuristic Controls on FLEURS) & Narrow Single-Model RTN Scope: While RAVDESS is evaluated across all 7 arms (FP16 + Selected, Front-block, and Stratified-uniform at 6 and 7 bits), FLEURS EN$\to$DE translation is evaluated on only 3 arms (FP16, Selected 6-bit, Selected 7-bit). At 6 and 7 bits per weight under per-channel RTN, Front-block and Stratified-uniform may also preserve near-FP16 BLEU/chrF on FLEURS. Without evaluating the heuristic controls on FLEURS, the paper does not demonstrate a genuine cross-endpoint trade-off (where the translation-selected allocation outperforms the controls on translation while underperforming on emotion); the selected allocation may simply be a noisy/suboptimal bit assignment overfit to the 387-example translation development split. Moreover, the empirical evidence is restricted to a single model (Qwen2-Audio-7B-Instruct), a rudimentary {4,8}\{4, 8\}-bit whole-module RTN scheme (omitting modern PTQ methods such as AWQ, GPTQ, SmoothQuant, or HQQ/Diet-KIT), and mild 6/7-bit budgets.
  2. Black-Box Opacity of the Bit Allocations Misses the Underlying Architectural Mechanism: Section 3.2 (line 211) states that serialized files define the 6- and 7-bit interventions, yet never reports which modules in Qwen2-Audio-7B-Instruct (Whisper-large-v3 audio encoder vs. audio projection adapter vs. Qwen-7B LLM decoder layers) were assigned 8 bits vs. 4 bits. Strikingly, Front-block 7-bit achieves 73.44%73.44\% on RAVDESS (+4.49%+4.49\% above Selected 7-bit, p=0.0020p = 0.0020, and +3.32%+3.32\% above FP16). Because Front-block upgrades blocks sequentially in model order, at a 7-bit budget (75%75\% of weights at 8 bits) it upgrades the entire Whisper audio encoder and early/middle LLM layers to 8 bits while leaving the final LLM decoder layers at 4 bits. Disclosing the module-level bit distribution across the audio encoder and text decoder for the selected and control plans is essential to explain why translation-selected plans degrade paralinguistic emotion accuracy.
  3. Unexplained Exclusion of One-Third of RAVDESS Speakers (N=16N = 16 of 2424), Unreported Non-Canonical Output Rates, and Prose/Table Inconsistencies:
    • Unexplained Subset Selection: Standard RAVDESS (Livingstone & Russo, 2018) contains 24 actors (12 male, 12 female) and 8 clips per non-neutral emotion (2 intensities ×\times 2 repetitions ×\times 2 sentences). The paper evaluates only Actors 1–16 and 4 clips per emotion without explaining why Actors 17–24 were excluded or which intensity/repetition subset was chosen.
    • Conflation of Formatting/Morphological Variants with Misclassification: The prompt mixes 6 adjectives (neutral, calm, happy, sad, angry, fearful), 1 noun (disgust), and 1 participle (surprised). The scorer adds a post-hoc rule mapping "disgusted" →\to "disgust" (line 190) but rejects other morphological variants ("surprise", "fear", "anger", "sadness") and never reports the non-canonical/unparseable response rate or per-emotion recall/confusion matrices across the 7 arms. Since the entire −3.71%-3.71\% drop at 6 bits is 19 clips (340340 vs. 359/512359/512), formatting shifts could drive a nontrivial share of the difference.
    • Prose vs. Table 3 Counting Error: Section 4.3 (page 6, lines 295–296) states for 7 - FP16 (%): "The 7-bit difference is negative for five actors, zero for eight, and positive for three." In Table 3 (page 7), 7 - FP16 (%) actually has 5 negative (Actors 01, 08, 09, 12, 13), 9 zero (Actors 02, 03, 04, 05, 06, 11, 14, 15, 16), and 2 positive (Actors 07, 10). Additionally, Table 4 (Appendix B) omits the Selected 7-bit minus FP16 contrast, and the Reproducibility Statement (line 496) references "complete manifests, allocation files... output hashes" that are missing from the manuscript.
Questions for authors
  1. FLEURS Control Performance (Clarification): What are the held-out FLEURS EN$\to$DE BLEU and chrF scores (and 95% CIs vs. FP16 and vs. Selected) for the 6-bit and 7-bit Front-block and Stratified-uniform controls? Do the translation-selected plans meaningfully outperform those heuristic controls on translation?
  2. Module-Level Bit Allocations (Clarification): Which specific modules in the Whisper-large-v3 audio encoder, projection adapter, and Qwen-7B LLM decoder received 8 bits vs. 4 bits in Selected 6/7-bit, Front-block 6/7-bit, and Stratified-uniform 6/7-bit? Does translation selection preferentially allocate 8-bit weights to the LLM text decoder at the expense of the audio encoder?
  3. RAVDESS Actor & Clip Selection (Clarification): Why were Actors 17–24 (8 of the 24 RAVDESS speakers) excluded, and how were the 4 clips per non-neutral emotion selected from the 8 available clips per actor (normal vs. strong intensity; repetition 1 vs. 2)? Does the 6-bit emotion deficit hold across all 24 speakers?
  4. Non-Canonical Outputs & Per-Emotion Breakdown (Clarification): Across the 7 RAVDESS arms, how many responses were scored as errors due to non-canonical formatting/morphology (e.g., "surprise", "fear", or conversational prefixes) versus valid canonical emotion confusions, and which emotion classes drive the −3.71%-3.71\% drop at 6 bits and the +4.49%+4.49\% advantage of 7-bit Front-block?
4 comments

The submitted PDF reports FLEURS only for FP16 and the two translation-selected plans; it contains no BLEU or chrF results for the front-block or stratified-uniform controls. Section 3.2 says selection used a separate 387-example development split but gives neither the objective nor module-by-module bit assignments. The text states whole-module 4/8-bit quantization, an 8-bit output head, and per-channel round-to-nearest; exact plans and implementation records are not included.

For RAVDESS, the paper specifies actors 1–16 and four clips per actor-emotion pair. It does not report why those actors were selected, which intensity/repetition combinations were retained, whether selection was fixed before evaluation, or results for all 24 actors. The scoring rule lowercases outputs, removes periods and commas, maps “disgusted” to “disgust,” and requires one canonical label. The PDF reports no non-canonical output counts or per-emotion confusion matrices, and raw predictions are unavailable. The 6-bit difference therefore cannot be decomposed into formatting errors and label confusions from the submitted record.

Section 4.3 miscounts the 7-bit actor differences: Table 3 contains five negative, nine zero, and two positive differences. Section 4.2 and Figure 2 report selected-7-bit minus FP16 as −1.17% [−2.73%, +0.20%], p=0.2656; Appendix Table 4 omits that comparison. The reproducibility statement names manifests, plans, hashes, and code, but the supplementary archive could not be retrieved for this check.

Thanks for the response, and for confirming the miscount in Section 4.3 and the missing comparison in Appendix Table 4. Unfortunately, none of the information I asked for is provided, and the manuscript has not been revised.

There are still no BLEU or chrF numbers for the four matched-budget controls on FLEURS (Q1). Without them, the paper cannot show that selecting on translation gives any translation advantage over controls that do as well or better on RAVDESS. The module-level bit assignments and the selection objective (Q2) are also still missing.

For Q3, there is no reason given for using actors 1-16 and the particular intensity/repetition subset, no statement on whether this choice was fixed before looking at the results, and no result on all 24 actors. This matters, since the 6-bit effect comes down to 19 clips (p=0.0148p = 0.0148). For Q4, the response says the raw predictions are not available, so we cannot tell how much of the 6-bit drop comes from formatting failures rather than wrong labels. That also contradicts the Reproducibility Statement, which lists manifests, allocation files and output hashes, and the response's remark that the supplementary archive "could not be retrieved".

The concerns of the other reviewers are unaddressed as well (FLEURS controls and reproduction details from 1zFC and 4bG6, motivation and novelty from 6y8Y and 4bG6). I am keeping my score at 2.

Thank you for laying this out. Our prior response did not answer Q1–Q4 or revise the manuscript. The submitted paper does not report matched-budget BLEU/chrF results on FLEURS, module-level bit assignments or the selection objective, the rationale or prespecification of the actor subset or results for all 24 actors, or invalid-output rates and confusion matrices. The supplementary archive was inaccessible in our check, so we could not verify the manifests, allocations, or output hashes named in the Reproducibility Statement. We leave these items unresolved and understand you are maintaining your score.

Thank you for pressing for the missing evidence. I completed the requested runs and revised the working manuscript source. The PDF attached to the submission remains the submitted version; this comment reports the new measurements and the changes in the revised draft.

FLEURS (same 508 utterances; SacreBLEU 2.5.1). BLEU/chrF scores: FP16 20.363/49.732; selected-6 20.216/49.600; front-6 and uniform-6 20.327/49.680; selected-7 20.327/49.680; front-7 and uniform-7 20.336/49.659. Control-minus-FP16 paired 95% intervals all include zero: at 6 bits, BLEU -0.036 [-0.385,+0.262], chrF -0.053 [-0.332,+0.189]; at 7 bits, BLEU -0.027 [-0.390,+0.291], chrF -0.073 [-0.361,+0.173]. Versus selected, both 6-bit controls have BLEU +0.111 [-0.001,+0.271] and chrF +0.079 [+0.004,+0.179]; both 7-bit controls have BLEU +0.009 [-0.140,+0.132] and chrF -0.021 [-0.108,+0.053]. Thus the selected plans show no consistent held-out translation advantage over these controls.

Selection and modules. Each selected plan maximizes mean chrF over seven predefined plans on the separate 387-example translation development split; RAVDESS was not used. The 418 planned modules are 192 audio-tower, one audio-projection, and 225 language-model modules (224 layer projections plus lm_head). All audio-tower, audio-projection, and lm_head weights are 8-bit. Counts of 4/8-bit language-model modules are selected-6 106/119, selected-7 49/176, either control-6 121/104, and either control-7 61/164. Exact decoder-layer masks below list the 0-based layer indices assigned 4-bit for each projection; all unlisted indices and projections are 8-bit. The prefix is language_model.layers; all omitted components above are 8-bit. S6: down 0-4,6,8,10,12-13,15-18,20,23-30; gate 0,5-6,9-13,15-18,22-26,28,30; up 0-2,6-12,14-17,19-20,22-23,25,27,30; k 1-2,7-8,15,31; o 0-2,8-9,11-14,16,25-26,28-29; q 8,29-30; v 0-2,4-5,8,10-11,14,17-18,20,22-27,29,31. S7: down 3-4,6,10,12-13,15,17; gate 5-6,9-11,22-26; up 6-12,14-15,17,19-20,22-23,25,27; o 0; v 0,2,4-5,8,11,14,17,23-24,26-27,29,31. F6: down/gate/up 0-9,11,24-28,30-31; k 0-9,11,25,27-28,30-31; o/q/v 0-9,11,25-28,30-31. U6: down/gate/up 0-9,11,17,24-26,28,30-31; k 0-9,11,24-25,28,30-31; o/q/v 0-9,11,24-26,28,30-31. F7: down/gate/up 0-2,4-6,8-9,28; k/q 0-2,4-6,8,28; o/v 0-2,4-6,8-9,28. U7: down/gate/up 0-2,4,6,8-9,28,31; k/q 0-2,4,6,8,28,31; o/v 0-2,4,6,8-9,28,31. The implementation uses symmetric per-channel round-to-nearest weight quantization followed by dequantized inference.

RAVDESS. The original 512 items are actors 1-16 at normal intensity, with both statements and both repetitions; an audio-path join confirms they are exactly included in the new 768-item, 24-actor set. The project record does not preserve why the initial analysis excluded actors 17-24 or establish whether that subset was fixed before outcome review. This extension is post hoc and overlaps the primary set. Across all seven 24-actor arms, 768/768 outputs per arm matched a canonical label and invalid count was zero. Accuracy and paired actor-bootstrap differences from FP16 are: FP16 69.79%; selected-6 66.80%, -2.99% [-4.95,-1.17]; front-6 67.97%, -1.82% [-3.13,-0.65]; uniform-6 68.49%, -1.30% [-2.60,-0.26]; selected-7 68.75%, -1.04% [-2.21,0.00]; front-7 72.79%, +2.99% [+1.82,+4.04]; uniform-7 69.92%, +0.13% [-0.78,+1.04]. Selected-6 remains lower, while selected-7 remains unresolved. Per-emotion recall for all arms is in the revised draft; all arms have 0.0% calm recall. Selected-6 fearful/neutral recall is 59.4%/59.4% versus 74.0%/69.8% for FP16; front-7 disgust/fearful recall is 53.1%/78.1% versus 37.5%/74.0% for FP16.

Full 24-actor confusion counts (true rows and predicted columns use A,C,D,F,H,N,S,U = angry, calm, disgust, fearful, happy, neutral, sad, surprised; each row lists nonzero predicted counts and omitted cells are zero): FP16: A:A92,H2,S2; C:A2,H5,N57,S32; D:A11,D36,S49; F:A3,F71,S21,U1; H:A6,H85,S1,U4; N:A5,H1,N67,S23; S:A1,N1,S94; U:A1,F1,H1,S2,U91 S6: A:A93,H1,S2; C:A3,H8,N58,S27; D:A10,D36,S50; F:A6,F57,H2,S29,U2; H:A5,H89,S1,U1; N:A4,H12,N57,S23; S:A1,N2,S93; U:A3,H3,S2,U88 F6: A:A93,H1,S2; C:H4,N56,S36; D:A16,D32,S48; F:A5,F64,S23,U4; H:A5,H84,S1,U6; N:A4,H1,N64,S27; S:N1,S95; U:A2,F1,H1,S2,U90 U6: A:A93,H1,S2; C:A1,H2,N56,S37; D:A12,D29,S55; F:A5,F69,S22; H:A5,H86,S1,U4; N:A4,H1,N64,S27; S:S96; U:A3,F1,H1,S2,U89 S7: A:A91,H1,S4; C:H2,N60,S34; D:A9,D35,S52; F:A5,F65,S26; H:A4,H86,S2,U4; N:A3,N66,S27; S:N1,S95; U:A2,F1,H1,S2,U90 F7: A:A92,D1,H1,S2; C:H3,N63,S30; D:A6,D51,S39; F:A2,F75,S18,U1; H:A2,H85,S2,U7; N:A1,H2,N70,S23; S:N2,S94; U:F1,H1,S2,U92 U7: A:A93,H1,S2; C:A1,H3,N57,S35; D:A8,D41,S47; F:A2,F73,S19,U2; H:A5,H83,S1,U7; N:A5,H2,N61,S28; S:N1,S95; U:A1,F1,H1,S2,U91

Reviewer_6y8YOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how quantization affects the performance of speech language models, specifically Qwen2-Audio. The authors evaluate two speech tasks: speech translation, which is evaluated using text-based metrics, and speech emotion recognition, where the emotion labels cannot be inferred from the transcript alone. Experimental results show that the quantized model achieves translation scores close to FP16, while the 6-bit configuration exhibits degraded emotion recognition performance. The authors also compare different quantization configurations with matched bit budgets. The paper concludes that text-based metrics alone are insufficient to evaluate quantized speech models on tasks involving non-lexical information.

Critical strengths

None

Critical weaknesses

1. Writing and presentation:

The writing is my primary concern with this paper. The paper is poorly organized and difficult to follow, with unclear logical connections between sentences and paragraphs. The Abstract, Introduction, and Related Work do not clearly explain the research motivation, research gap, or main contributions. The authors frequently introduce technical details and statistical terminology without sufficient high-level context. The paper also contains many repetitive and overly defensive explanations about what the results cannot establish, which further obscure the main findings and scientific message. As a result, I found it difficult to understand the purpose of the experimental design and what the authors want to convey through their results.

2. Unclear novelty and limited experimental scope:

The authors repeatedly emphasize that text-based metrics cannot establish performance on non-lexical speech tasks. While this is true, I do not see why this is a meaningful research gap. The paper does not identify prior work that uses text-based metrics to claim preserved performance on non-lexical tasks after quantization. Reporting ASR or speech translation performance does not necessarily imply that a model preserves all speech understanding capabilities.

Furthermore, speech language models can perform a broad range of speech and audio understanding tasks. I do not see how evaluating only speech translation and emotion recognition on a single model provides meaningful insights into the effects of quantization on these capabilities. Well-established benchmarks, such as Dynamic-SUPERB and AIR-Bench, already cover a much broader range of speech and audio tasks and could support a more comprehensive evaluation. However, the paper does not explain why these existing benchmarks were not considered or why a separate evaluation protocol is necessary, especially when the experiments themselves use existing datasets.

Questions for authors
  1. Could the authors clarify the specific research gap that motivates this work? In particular, the paper does not identify prior studies that use text-based metrics to claim preserved performance on non-lexical speech tasks after quantization. It is unclear what problem in existing evaluation practices the authors aim to address.

  2. Could the authors clarify what new insights the proposed evaluation protocol provides beyond evaluating individual speech tasks with their corresponding metrics? I found it difficult to identify the main contribution of the proposed protocol and how it advances our understanding of speech model quantization.

7 comments

The paper asks whether translation scores after quantization provide evidence about an endpoint whose labels cannot be recovered from the transcript. It cites LISTEN's cue-conflict work and studies a different inference using the same selected allocations on translation and acted emotion recognition. It does not attribute to prior work a claim that translation parity guarantees emotion performance.

The evidence is conditional on one Qwen2-Audio checkpoint and one RAVDESS subset. The same two sentences occur with all eight labels, so sentence identity is non-diagnostic; this balance does not establish that emotion accuracy comes from acoustic cues. FLEURS differences from FP16 are unresolved at both budgets. On RAVDESS, the selected 6-bit allocation is lower than FP16 by 3.71% with a speaker interval excluding zero; the 7-bit difference is −1.17% with an interval containing zero. The study shows that a translation result leaves the other endpoint unmeasured; it does not establish a recurring quantization effect.

Dynamic-SUPERB and AIR-Bench are not evaluated, so the experiments do not demonstrate broader task coverage or an advantage over a multi-task benchmark. The contribution is a single-checkpoint, two-endpoint case study.

Thank you for the clarification. However, the response does not sufficiently address my main concerns regarding the research motivation, novelty, limited experimental scope, and writing quality. I will maintain my original assessment and rating.

AuthorsOpenReview ↗

Response on scope and added evaluations

Thank you for specifying the unresolved issues. The central question is narrower than a transfer hypothesis: we ask whether a text-task score provides evidence about a separate speech endpoint whose label is not recoverable from the transcript. We do not attribute to prior work a statement that translation parity guarantees emotion preservation. LISTEN studies cue conflict; our contribution is a bounded quantization case study applying the same fixed precision interventions across translation and acted emotion. It is not a replacement for Dynamic-SUPERB or AIR-Bench, which we did not evaluate.

We completed the four matched-control FLEURS runs on the same 508 utterances. Scores (BLEU/chrF): FP16 20.363/49.732; selected-6 20.216/49.600; front-block and uniform-6 20.327/49.680; selected-7 20.327/49.680; front-block and uniform-7 20.336/49.659. Control-minus-FP16 paired 95% intervals include zero. At 6 bits, controls exceed selected by +0.111 BLEU [-0.001,+0.271] and +0.079 chrF [+0.004,+0.179]; at 7 bits, differences are +0.009 BLEU [-0.140,+0.132] and -0.021 chrF [-0.108,+0.053]. These results show no consistent translation advantage for the selected maps, and we removed any interpretation as an empirically demonstrated cross-task trade-off or selection benefit.

We also extended RAVDESS post hoc to all 24 actors. Selected-6 accuracy is 66.80% versus 69.79% for FP16, an absolute difference of -2.99% [-4.95%,-1.17%]; selected-7 is -1.04% [-2.21%,0.00%]. Because this extension contains the original 16 actors, it is overlapping and not an independent replication. Across seven arms, 768/768 outputs per arm matched a canonical label (zero invalid); the revised paper adds per-emotion recall and full confusion counts. The primary 16-actor selected-6 result remains -3.71% [-6.05%,-1.37%], p=0.0148; the 7-bit result remains unresolved.

We revised the introduction and discussion to state the specific inferential gap, distinguish this case study from broad benchmark coverage, move design details after the research question, and remove repeated caveats. The evidence does not establish a recurring effect across checkpoints or identify which audio cues the model uses; the revision states those boundaries directly. The revised PDF has been compiled from the final source, but is not yet attached to OpenReview, so these text changes are not visible in the current submission PDF.

Reviewer_1zFCOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates fixed mixed-precision allocations for Qwen2-Audio on speech translation and emotion recognition. Translation scores remain close to FP16, while the selected 6-bit allocation reduces emotion-recognition accuracy by 3.71 percentage points. The study argues that translation scores alone cannot establish performance on tasks requiring nonlexical vocal cues.

Critical strengths

The evaluation makes a useful practical point. Testing the same allocations on translation and sentence-balanced emotion recognition shows why audio models need evaluation beyond text-output metrics.

The statistical analysis is careful. Matched-budget controls and paired speaker-level comparisons strengthen the results, and the authors correctly distinguish inconclusive differences from evidence of preserved performance.

Critical weaknesses

The contribution is narrow. Results from one checkpoint, two selected allocations and one acted-emotion dataset do not establish whether the observed degradation is a recurring quantization effect or specific to these configurations.

Key reproduction details are missing. The conclusions depend on specific allocation files, but the manuscript does not provide their module-to-bit assignments or accessible identifiers. The development-set procedure used to select these allocations also needs clarification.

Questions for authors

Could you provide the exact module-wise bit assignments, clarify which components are quantized and included in the parameter-weighted budget, and specify the per-channel round-to-nearest implementation? How were the two plans selected using the 387-example development set, and what objective guided selection?

Could you report FLEURS BLEU and chrF for all four matched-budget controls? Do the selected plans offer a translation advantage over these controls, or do the controls achieve comparable translation scores alongside their higher RAVDESS point estimates?

4 comments

The manuscript reports three FLEURS arms: FP16 and the two translation-selected allocations. It does not report BLEU or chrF for the four matched-budget controls, so the submitted results cannot establish that either selected plan outperforms those controls on translation. RAVDESS controls are a separate endpoint and do not answer the FLEURS question.

Section 3.2 states that the selected plans were fixed on a separate 387-example translation development split, initialize whole modules at 4 bits, keep the output head at 8 bits, and upgrade whole modules to 8 bits under a parameter-weighted budget using dequantized per-channel round-to-nearest. The PDF does not list module-by-module assignments or the objective that selected the plans. The exact allocation files and selection record are needed to reproduce the plan choice.

Thank you for identifying the missing translation controls. The revised PDF and source supplement are now attached.

On the same 508 English-to-German FLEURS test utterances (SacreBLEU 2.5.1), BLEU/chrF are: FP16 20.363/49.732; selected-6 20.216/49.600; front-block-6 and uniform-6 20.327/49.680; selected-7 20.327/49.680; and front-block-7 and uniform-7 20.336/49.659. Paired 95% intervals for each control versus FP16 include zero. Relative to the selected allocation, each 6-bit control is +0.111 BLEU [-0.001, +0.271] and +0.079 chrF [+0.004, +0.179]; each 7-bit control is +0.009 BLEU [-0.140, +0.132] and -0.021 chrF [-0.108, +0.053]. The selected maps therefore show no consistent held-out translation advantage over the matched-budget controls.

The selection criterion is maximum mean chrF across seven specified allocation plans on the separate 387-example development split; RAVDESS was not used. The revised manuscript states the budget and precision rule, reports module counts, and lists the decoder-layer/projection masks for each plan in the appendix. It also specifies symmetric per-channel round-to-nearest weight quantization followed by dequantized inference. The results support a bounded comparison on these two tasks and this checkpoint; they do not show that the selected allocation is best on either endpoint.

The new FLEURS runs answer my main experimental question. At 6 bits the matched controls actually score slightly higher on chrF, and at 7 bits the differences are small. That makes the result easier to interpret: choosing a plan on translation gives no consistent advantage on held-out translation here, and translation scores alone miss the drop in emotion accuracy. The selection objective and module-level masks also address my reproducibility concern. I would raise my rating from 1 to 2. The remaining limitation is scope. The 24-actor result adds confidence in the 6-bit finding, but it includes the original 16 actors and still concerns one checkpoint on one acted-emotion corpus.

Thank you for updating your assessment and for noting that the matched controls and module-level masks address your experimental and reproducibility questions. We agree that the 24-actor analysis is overlapping: it includes the original 16 actors, so it is a sensitivity analysis, not an independent replication. It provides a broader within-dataset check but does not establish generalization across checkpoints or acted-emotion corpora. Our conclusions remain limited to this Qwen2-Audio checkpoint and RAVDESS. We appreciate your recognizing the remaining scope limitation alongside the new evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.