- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the “perception-override reflex”: speech language models sometimes return an audio-supported attribute even when instructed to select the alternative. The authors introduce OVERRIDE by reformulating 4,204 existing MMSU clips across 40 tasks into matched affirmative, negated, and text-informed conditions. Across seven models, negated instructions cause substantial failures even on affirmative-correct items. Further analysis across six models and 56 phrasings shows that text-side instruction executability correlates with susceptibility under audio, while leaving substantial residual variation unexplained.
Motivated by these findings, the authors apply on-policy self-distillation using a frozen teacher supplied with the ground-truth attribute in text and a reliably executable canonical instruction. On Qwen2.5-Omni, conditional negated accuracy improves from 63.7% to 83.6%, and joint affirmative–negated accuracy improves from 45.4% to 60.9%. Similar improvements occur on Qwen3-Omni. The principal contributions are the controlled evaluation protocol, the characterization of phrasing-dependent failures, and an analysis-motivated mitigation.
Critical strengths
-
The controlled evaluation establishes a meaningful behavioral failure beyond ordinary task errors. Affirmative-correct filtering, matched text controls, and complementary joint accuracy provide substantially stronger evidence than simply reporting lower accuracy on negated questions. What I found striking is that a model can answer the affirmative question correctly and execute the negated instruction in the text control, yet still fail when the two abilities have to work together on audio. The paired evaluation makes this disconnect particularly clear. Replication across models and phrasings, together with four-option and probability-based analyses, strengthens the observation.
-
The mitigation produces substantial gains, and the ablations connect the analysis to an actionable design choice. Improvements in both conditional and joint accuracy show that the gains are not solely an artifact of conditional evaluation. Table 2 supports the usefulness of text conditioning and canonical phrasing within the OPSD setup: on Qwen2.5-Omni, pooled conditional accuracy increases from 75.7% with OPSD-audio to 78.3% with noncanonical OPSD-text and 83.6% with the full method. I like the simplicity of the mitigation and its direct connection to the preceding analysis. The teacher-phrasing experiment in Table 3 was particularly interesting to me: changing the wording given to the teacher can turn a helpful training signal into one that substantially worsens the student.
Critical weaknesses
-
The scope of the empirical contribution and the mitigation’s advantage over closely related approaches need clearer establishment. Speech-IFEval already motivates separating speech perception from instruction execution and examines sensitivity to instruction variations, while Semantic Inversion, Identical Replies studies failures under negated queries despite correct responses to positive queries. OVERRIDE adds a useful matched speech-negation protocol, but the text control directly supplies the ground-truth intermediate attribute, changing both modality and information accessibility. The controls therefore establish the behavioral failure without determining how specifically it concerns negation, rather than a broader difficulty in composing perception with answer transformations. The paper appropriately acknowledges that its analysis is not mechanistic; my concern is the scope of the additional empirical insight. For the mitigation, the paper explicitly adopts an existing OPSD objective, and DeSTA2.5-Audio provides a closely related precedent for using a backbone LLM’s responses to textual descriptions of audio as supervision for audio-conditioned inputs. The reported ablations support the teacher-condition choices within OPSD. These results left me with a fairly specific question: once the teacher is given the correct attribute and a reliably executable instruction, how much additional benefit comes from learning on the student’s own trajectories? A comparison with offline responses from that same teacher would help me judge what the on-policy component contributes. Such a comparison should hold the backbone, training examples, teacher information, and trainable parameters fixed. Gold-answer SFT is useful, but it does not isolate this question because it also changes the supervision source. The inference comparison is similarly narrow. The cited negation literature evaluates task decomposition and self-refinement, whereas the present verbalize-then-answer baseline generates two lines within one audio-conditioned response. This does not evaluate a separate perception stage followed by text-only instruction execution. These omissions limit conclusions about the intervention’s comparative value, although they do not invalidate the reported behavioral findings or the gains over the evaluated baselines.
-
The mitigation evaluation leaves important uncertainties about data generalization and the Qwen3 SFT comparison. Appendix F reports 11,400 override and 11,619 affirmative training examples, but does not sufficiently document their source datasets, splits, or audio-level separation from OVERRIDE evaluation recordings. Disjoint question phrasings do not establish disjoint audio. I am not asserting that overlap occurred. This was the point where I found it hardest to interpret the mitigation results: I could not tell whether the model had learned to handle new recordings or new questions about recordings it had already encountered during training. Both are worth studying, but they support different conclusions about generalization. Appendix F also states that Qwen3 distillation variants are trained for one epoch, whereas SFT is trained for half an epoch. Different training durations can be justified, but the manuscript does not explain the validation or checkpoint-selection procedure sufficiently to establish that the reported SFT result is a competitive operating point. This matters because its affirmative accuracy falls from 77.0% to 65.6%, and its joint accuracy falls from 49.1% to 44.3%. The full method demonstrably preserves affirmative accuracy in the reported experiment, but its comparative advantage over SFT is harder to attribute to the training objective without a justified tuning procedure. This concern specifically applies to Qwen3; the manuscript states that Qwen2.5 SFT uses the same training schedule as distillation.
Questions for authors
-
Are the mitigation training recordings and segments disjoint from OVERRIDE evaluation audio? Please identify the source datasets, splits, and overlap checks for the override and affirmative training groups, and clarify which data were used to select the canonical teacher phrasing. Verified audio separation would resolve an important uncertainty. If evaluation recordings are reused with different questions, the corresponding results should be described as generalization across phrasings on seen audio. Please also clarify the retention-data splits: 4.3 mentions LibriSpeech dev-clean transcripts, whereas Appendix F refers to training-split utterances.
-
What supports the choice of the reported Qwen3 SFT checkpoint? Why was SFT trained for half an epoch, and how were its learning rate and training duration selected? Do the affirmative- and joint-accuracy disadvantages persist under a justified validation and checkpoint-selection procedure? I am interested in whether the reported comparison reflects a competitive SFT configuration, rather than requiring identical epoch counts across objectives.
-
How specific is the observed failure to negated instructions? A matched audio-conditioned control requiring a non-negation answer remapping, with comparable task complexity and output constraints, would help distinguish negation-related difficulty from a broader problem in combining perception with answer transformations. Is there evidence from such a control that sharpens the scope of the behavioral claim?
-
Does the mitigation retain an advantage over relevant adapted alternatives? The most informative training comparison would use offline responses from the same privileged canonical teacher, following the transferable self-generated-supervision idea in DeSTA. For inference, an explicit two-stage pipeline—predicting the attribute from audio and then executing the instruction in a separate text-only context—would provide a useful operational baseline when evaluated over the full item set with its additional cost reported. On affirmative-correct items, the predicted attribute already equals the gold attribute, so this pipeline should primarily be treated as an end-to-end alternative rather than a new diagnostic beyond the existing text control. A representative self-refinement comparison would also help assess the breadth of the prompting conclusion.
5 comments
Weakness 2 & Question 2 - SFT Baseline Comparability Thank you for raising this concern. The submitted Qwen3 SFT results were obtained after half an epoch of training. We have now completed a full-epoch SFT run and evaluated OPSD-text at half an epoch for comparison. Neither method used checkpoint selection; all results are from the final checkpoint of each run. The results below use the corrected Qwen3-Omni evaluation described in our general response.
| Qwen3-Omni-30B-A3B | epochs | Ia | Acc|Ia | Reflex | Joint | Text ctrl |
|---|---|---|---|---|---|---|
| Base | – | 77.1 | 64.5 | 32.8 | 49.7 | 86.3 |
| SFT | 0.5 | 71.8 | 76.7 | 21.7 | 55.0 | 87.2 |
| SFT | 1.0 | 75.1 | 79.7 | 18.6 | 59.8 | 94.0 |
| OPSD-audio | 1.0 | 78.8 | 76.0 | 22.0 | 59.9 | 91.2 |
| OPSD-text | 0.5 | 79.2 | 77.3 | 20.3 | 61.2 | 91.8 |
| OPSD-text | 1.0 | 78.9 | 82.1 | 16.2 | 64.8 | 96.3 |
OPSD-text consistently outperforms SFT at the same number of epochs, with joint accuracy improvements of 6.2 points at 0.5 epoch and 5.0 points at 1 epoch. Moreover, SFT reduces affirmative accuracy relative to the base model, while OPSD-text preserves it. Both methods use the same LoRA configuration (rank 64, α = 128) and learning rate (1e-4). Neither method used validation-based checkpoint selection. An additional SFT run with a lower learning rate (2e-5, 0.5 epoch) achieved 55.8% Joint accuracy, compared with 55.0% at 1e-4. We will update Table 2 with the full-epoch SFT results and clarify the training settings in Appendix F.
Weakness 1 & Question 4 — Additional Baseline Comparisons
Thank you for the suggestions. We conducted all three additional comparisons on Qwen2.5-Omni-7B using the full benchmark.
Offline distillation with the same teacher. We generated offline answers using the same frozen text-conditioned teacher as OPSD-text and trained the student with the SFT recipe, keeping the backbone, training examples, and trainable parameters fixed.
| Supervision | Acc|Ia | Reflex | Joint |
|---|---|---|---|
| Gold labels (SFT) | 75.5 | 22.8 | 52.9 |
| Offline teacher | 75.3 | 21.9 | 54.8 |
| On-policy teacher (OPSD-text) | 83.6 | 14.5 | 60.9 |
The offline teacher agrees with the gold labels on 98.1% of training examples, and offline distillation performs similarly to SFT. In contrast, OPSD-text improves Joint accuracy by 6.1 points over this offline baseline, supporting the benefit of on-policy distillation.
Two-stage inference. We implemented a two-stage pipeline: Stage 1 predicts the attribute from audio using the affirmative question, and Stage 2 executes the negated instruction in a separate text-only call using the predicted attribute.
| Method | Calls | Acc|Ia | Reflex | Joint | Acc (all items) |
|---|---|---|---|---|---|
| Base, single-call | 1 | 63.7 | 31.1 | 45.4 | 59.2 |
| Base, two-stage | 2 | 91.5 | 5.0 | 65.2 | 67.7 |
| OPSD-text, single-call (Seed 1) | 1 | 83.5 | 14.4 | 60.9 | 71.8 |
| OPSD-text, two-stage | 2 | 97.1 | 1.5 | 70.7 | 71.8 |
Two-stage inference is a strong alternative, improving the base model’s Acc|Ia from 63.7% to 91.5% and Joint accuracy from 45.4% to 65.2%. Combining it with OPSD-text further improves Acc|Ia to 97.1% and Joint to 70.7%. However, errors in the first-stage attribute prediction propagate to the second stage, leaving the overall accuracy of OPSD-text unchanged at 71.8% despite the improvement in conditional accuracy. Moreover, two-stage inference requires explicitly decomposing the task and making two model calls, whereas OPSD-text improves single-call performance without changing the inference pipeline.
Self-refinement. We evaluated one iteration of self-refinement (answer, critique, refine), following Madaan et al., with audio provided in all three calls.
| Method | Calls | Acc|Ia | Reflex | Joint |
|---|---|---|---|---|
| Direct prompting | 1 | 63.7 | 31.1 | 45.4 |
| Self-refinement | 3 | 43.0 | 50.9 | 30.6 |
| OPSD-text | 1 | 83.6 | 14.5 | 60.9 |
Self-refinement substantially reduces accuracy. We observed that the critique stage rejects the initial answer approximately 90% of the time, regardless of its correctness, often causing the model to return the audio-supported attribute instead of following the negated instruction.
We will include these comparisons in Section 4.
We sincerely thank the reviewer again for the thorough and constructive feedback. Your suggestions have been particularly helpful in strengthening our evaluation.
We would like to kindly follow up on our rebuttal. We have conducted the additional experiments you suggested, including offline distillation, two-stage inference, self-refinement, and non-negation remapping, as well as clarifying the training–evaluation separation and SFT baseline comparability.
We would appreciate hearing whether these results have addressed your main concerns. If so, we would be grateful if you could consider revisiting your score.
We sincerely thank the reviewer for their thorough and thoughtful feedback. We found the comments insightful and constructive, and several suggestions inspired us to conduct additional experiments that helped strengthen our work.
Weakness 1 & Question 3 - Empirical contribution and scope Thank you for the detailed feedback. We agree that prior work has studied instruction-following failures in speech LLMs and negation failures in text-based LLMs. However, our work investigates a different question.
Our work goes beyond this setting by showing that speech LLMs frequently fail on negated instructions under audio, even when they can correctly recognize the audio attribute and follow the same negated instructions when the attribute is provided in text. For example, Qwen2.5-Omni achieves 91.5% accuracy on the text controls but only 63.7% under audio. This reveals a behavioral gap between audio-conditioned and text-conditioned instruction following, even on items where the model correctly identifies the audio attribute under affirmative instructions. We agree that the text control provides direct access to the correct attribute. Our claim is behavioral: even when the model can identify the attribute from audio and execute the instruction with the attribute provided in text, it frequently fails when both are required in a single audio-conditioned response.
Beyond identifying this gap, we systematically analyze 56 instruction phrasings across six models and show that text-side instruction accuracy predicts the frequency of errors under audio. These findings further guide our mitigation design.
Non-negation answer remapping. Our original evaluation already covers multiple forms of semantic negation, including held-out phrasings without explicit negation words (H1–H10, Table 7). Following the suggestion, we further evaluated Qwen2.5-Omni on the same 4,204 items using a non-negation code-word mapping task. For example, the model is instructed to identify the speaker’s gender and answer “alpha” for male or “beta” for female. All results are evaluated on items answered correctly under the affirmative instruction.
While both tasks achieve high text-control accuracy (97.0% for negation and 98.2% for remapping), negation performs substantially worse under audio (69.7% vs. 84.8% Acc|Ia). This shows that the audio-conditioned difficulty is more pronounced for negation in this comparison. Moreover, the two tasks do not necessarily share the same failure pattern: our reflex specifically refers to models returning the perceived attribute when instructed to exclude it, whereas the underlying causes of remapping errors remain unclear.
Weakness 2 & Question 1 - Training and Evaluation Separation Thank you for raising this concern. We confirm that the training and evaluation audio are disjoint. We address the specific questions below.
Training data and audio separation. Our training data contain 16,590 unique audio clips. The two main training groups are:
OVERRIDE evaluation uses 4,204 audio clips from MMSU, whose source datasets are different from those used for training. We checked the source datasets and filenames and found no overlap between training and evaluation audio. Therefore, the mitigation results evaluate generalization to unseen recordings, not just unseen instruction phrasings.
Canonical teacher phrasing. T1 was selected using training data only. It achieved 100% text-control accuracy on 1,000 training override items, where the correct attribute was provided in text. The 93 in Table 3 was measured afterwards on the OVERRIDE phrasing subset for the analysis in §3; it was not used for selection. We will clarify this in Appendix F.
Retention data split. The transcription retention data use LibriSpeech dev-clean (2,703 utterances), while WER is evaluated on test-clean (2,620 utterances). These splits have disjoint speakers. The reference to “training-split utterances” in Appendix F is a typo, which we will correct.
We will add the dataset sources, splits, and overlap checks to Appendix F.