acceptodds
Under review as a conference paper at ICLR 2027

The Perception-Override Reflex: When Speech Language Models Follow Perception Over Instructions

Abstract

Speech language models have become increasingly capable of recognizing paralinguistic information such as gender, emotion, and accent from audio. However, we find that their performance degrades sharply when an affirmative instruction is replaced by its negated counterpart, with models frequently returning the option explicitly excluded by the instruction when that option is supported by the audio. We term this failure mode the perception-override reflex. To characterize it, we introduce Override, a controlled benchmark that evaluates 4,204 audio clips under matched affirmative, negated, and text-control variants, allowing us to control for affirmative-task errors and text-side instruction execution. Across seven speech LLMs, negated instructions induce substantial accuracy drops even after restricting evaluation to items answered correctly under the corresponding affirmative instruction. We then ask what predicts variation in the reflex. Across 56 phrasings, text-side executability consistently predicts susceptibility under audio across six models, but does not fully account for the observed behavior: substantial reflex remains for highly executable instructions, and counterfactual formulations exhibit additional difficulty beyond that captured by text-side executability. These findings motivate on-policy self-distillation from the frozen model conditioned on the attribute in text and a reliably executable canonical phrasing. On Qwen2.5-Omni, this raises aggregate negated-instruction accuracy from 63.7% to 83.6% without degrading affirmative audio performance. Together, our results identify a reproducible failure in combining auditory evidence with negated instruction following, and show that text-side instruction competence provides both a predictor of vulnerability and a useful signal for mitigation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
speech language models, negation, multimodal instruction following, audio understanding, robustness, self-distillation, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rDg3OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the “perception-override reflex”: speech language models sometimes return an audio-supported attribute even when instructed to select the alternative. The authors introduce OVERRIDE by reformulating 4,204 existing MMSU clips across 40 tasks into matched affirmative, negated, and text-informed conditions. Across seven models, negated instructions cause substantial failures even on affirmative-correct items. Further analysis across six models and 56 phrasings shows that text-side instruction executability correlates with susceptibility under audio, while leaving substantial residual variation unexplained.

Motivated by these findings, the authors apply on-policy self-distillation using a frozen teacher supplied with the ground-truth attribute in text and a reliably executable canonical instruction. On Qwen2.5-Omni, conditional negated accuracy improves from 63.7% to 83.6%, and joint affirmative–negated accuracy improves from 45.4% to 60.9%. Similar improvements occur on Qwen3-Omni. The principal contributions are the controlled evaluation protocol, the characterization of phrasing-dependent failures, and an analysis-motivated mitigation.

Critical strengths
  1. The controlled evaluation establishes a meaningful behavioral failure beyond ordinary task errors. Affirmative-correct filtering, matched text controls, and complementary joint accuracy provide substantially stronger evidence than simply reporting lower accuracy on negated questions. What I found striking is that a model can answer the affirmative question correctly and execute the negated instruction in the text control, yet still fail when the two abilities have to work together on audio. The paired evaluation makes this disconnect particularly clear. Replication across models and phrasings, together with four-option and probability-based analyses, strengthens the observation.

  2. The mitigation produces substantial gains, and the ablations connect the analysis to an actionable design choice. Improvements in both conditional and joint accuracy show that the gains are not solely an artifact of conditional evaluation. Table 2 supports the usefulness of text conditioning and canonical phrasing within the OPSD setup: on Qwen2.5-Omni, pooled conditional accuracy increases from 75.7% with OPSD-audio to 78.3% with noncanonical OPSD-text and 83.6% with the full method. I like the simplicity of the mitigation and its direct connection to the preceding analysis. The teacher-phrasing experiment in Table 3 was particularly interesting to me: changing the wording given to the teacher can turn a helpful training signal into one that substantially worsens the student.

Critical weaknesses
  1. The scope of the empirical contribution and the mitigation’s advantage over closely related approaches need clearer establishment. Speech-IFEval already motivates separating speech perception from instruction execution and examines sensitivity to instruction variations, while Semantic Inversion, Identical Replies studies failures under negated queries despite correct responses to positive queries. OVERRIDE adds a useful matched speech-negation protocol, but the text control directly supplies the ground-truth intermediate attribute, changing both modality and information accessibility. The controls therefore establish the behavioral failure without determining how specifically it concerns negation, rather than a broader difficulty in composing perception with answer transformations. The paper appropriately acknowledges that its analysis is not mechanistic; my concern is the scope of the additional empirical insight. For the mitigation, the paper explicitly adopts an existing OPSD objective, and DeSTA2.5-Audio provides a closely related precedent for using a backbone LLM’s responses to textual descriptions of audio as supervision for audio-conditioned inputs. The reported ablations support the teacher-condition choices within OPSD. These results left me with a fairly specific question: once the teacher is given the correct attribute and a reliably executable instruction, how much additional benefit comes from learning on the student’s own trajectories? A comparison with offline responses from that same teacher would help me judge what the on-policy component contributes. Such a comparison should hold the backbone, training examples, teacher information, and trainable parameters fixed. Gold-answer SFT is useful, but it does not isolate this question because it also changes the supervision source. The inference comparison is similarly narrow. The cited negation literature evaluates task decomposition and self-refinement, whereas the present verbalize-then-answer baseline generates two lines within one audio-conditioned response. This does not evaluate a separate perception stage followed by text-only instruction execution. These omissions limit conclusions about the intervention’s comparative value, although they do not invalidate the reported behavioral findings or the gains over the evaluated baselines.

  2. The mitigation evaluation leaves important uncertainties about data generalization and the Qwen3 SFT comparison. Appendix F reports 11,400 override and 11,619 affirmative training examples, but does not sufficiently document their source datasets, splits, or audio-level separation from OVERRIDE evaluation recordings. Disjoint question phrasings do not establish disjoint audio. I am not asserting that overlap occurred. This was the point where I found it hardest to interpret the mitigation results: I could not tell whether the model had learned to handle new recordings or new questions about recordings it had already encountered during training. Both are worth studying, but they support different conclusions about generalization. Appendix F also states that Qwen3 distillation variants are trained for one epoch, whereas SFT is trained for half an epoch. Different training durations can be justified, but the manuscript does not explain the validation or checkpoint-selection procedure sufficiently to establish that the reported SFT result is a competitive operating point. This matters because its affirmative accuracy falls from 77.0% to 65.6%, and its joint accuracy falls from 49.1% to 44.3%. The full method demonstrably preserves affirmative accuracy in the reported experiment, but its comparative advantage over SFT is harder to attribute to the training objective without a justified tuning procedure. This concern specifically applies to Qwen3; the manuscript states that Qwen2.5 SFT uses the same training schedule as distillation.

Questions for authors
  1. Are the mitigation training recordings and segments disjoint from OVERRIDE evaluation audio? Please identify the source datasets, splits, and overlap checks for the override and affirmative training groups, and clarify which data were used to select the canonical teacher phrasing. Verified audio separation would resolve an important uncertainty. If evaluation recordings are reused with different questions, the corresponding results should be described as generalization across phrasings on seen audio. Please also clarify the retention-data splits: 4.3 mentions LibriSpeech dev-clean transcripts, whereas Appendix F refers to training-split utterances.

  2. What supports the choice of the reported Qwen3 SFT checkpoint? Why was SFT trained for half an epoch, and how were its learning rate and training duration selected? Do the affirmative- and joint-accuracy disadvantages persist under a justified validation and checkpoint-selection procedure? I am interested in whether the reported comparison reflects a competitive SFT configuration, rather than requiring identical epoch counts across objectives.

  3. How specific is the observed failure to negated instructions? A matched audio-conditioned control requiring a non-negation answer remapping, with comparable task complexity and output constraints, would help distinguish negation-related difficulty from a broader problem in combining perception with answer transformations. Is there evidence from such a control that sharpens the scope of the behavioral claim?

  4. Does the mitigation retain an advantage over relevant adapted alternatives? The most informative training comparison would use offline responses from the same privileged canonical teacher, following the transferable self-generated-supervision idea in DeSTA. For inference, an explicit two-stage pipeline—predicting the attribute from audio and then executing the instruction in a separate text-only context—would provide a useful operational baseline when evaluated over the full item set with its additional cost reported. On affirmative-correct items, the predicted attribute already equals the gold attribute, so this pipeline should primarily be treated as an end-to-end alternative rather than a new diagnostic beyond the existing text control. A representative self-refinement comparison would also help assess the breadth of the prompting conclusion.

5 comments

We sincerely thank the reviewer for their thorough and thoughtful feedback. We found the comments insightful and constructive, and several suggestions inspired us to conduct additional experiments that helped strengthen our work.


Weakness 1 & Question 3 - Empirical contribution and scope Thank you for the detailed feedback. We agree that prior work has studied instruction-following failures in speech LLMs and negation failures in text-based LLMs. However, our work investigates a different question.

  • Speech-IFEval evaluates instruction following using constraints that do not depend on the audio content. In contrast, OVERRIDE studies instructions whose correct answers depend on the attribute perceived from audio.
  • Semantic Inversion, Identical Replies studies negation failures in text-only LLMs, where models answer positive questions correctly but fail on their negated counterparts.

Our work goes beyond this setting by showing that speech LLMs frequently fail on negated instructions under audio, even when they can correctly recognize the audio attribute and follow the same negated instructions when the attribute is provided in text. For example, Qwen2.5-Omni achieves 91.5% accuracy on the text controls but only 63.7% under audio. This reveals a behavioral gap between audio-conditioned and text-conditioned instruction following, even on items where the model correctly identifies the audio attribute under affirmative instructions. We agree that the text control provides direct access to the correct attribute. Our claim is behavioral: even when the model can identify the attribute from audio and execute the instruction with the attribute provided in text, it frequently fails when both are required in a single audio-conditioned response.

Beyond identifying this gap, we systematically analyze 56 instruction phrasings across six models and show that text-side instruction accuracy predicts the frequency of errors under audio. These findings further guide our mitigation design.

Non-negation answer remapping. Our original evaluation already covers multiple forms of semantic negation, including held-out phrasings without explicit negation words (H1–H10, Table 7). Following the suggestion, we further evaluated Qwen2.5-Omni on the same 4,204 items using a non-negation code-word mapping task. For example, the model is instructed to identify the speaker’s gender and answer “alpha” for male or “beta” for female. All results are evaluated on items answered correctly under the affirmative instruction.

While both tasks achieve high text-control accuracy (97.0% for negation and 98.2% for remapping), negation performs substantially worse under audio (69.7% vs. 84.8% Acc|Ia). This shows that the audio-conditioned difficulty is more pronounced for negation in this comparison. Moreover, the two tasks do not necessarily share the same failure pattern: our reflex specifically refers to models returning the perceived attribute when instructed to exclude it, whereas the underlying causes of remapping errors remain unclear.


Weakness 2 & Question 1 - Training and Evaluation Separation Thank you for raising this concern. We confirm that the training and evaluation audio are disjoint. We address the specific questions below.

Training data and audio separation. Our training data contain 16,590 unique audio clips. The two main training groups are:

  • Override (11,400 examples; 6,939 clips): LibriTTS-R (train-clean-100/360, train-other-500), AudioSet (unbalanced-train and strong-label), CREMA-D, TACOS, MagnaTagATune, Clotho v2 (development), and MusicCaps.
  • Affirmative (11,619 examples; 5,815 clips): The same source datasets and splits, with 2,628 clips shared with the override group.

OVERRIDE evaluation uses 4,204 audio clips from MMSU, whose source datasets are different from those used for training. We checked the source datasets and filenames and found no overlap between training and evaluation audio. Therefore, the mitigation results evaluate generalization to unseen recordings, not just unseen instruction phrasings.

Canonical teacher phrasing. T1 was selected using training data only. It achieved 100% text-control accuracy on 1,000 training override items, where the correct attribute was provided in text. The 93 in Table 3 was measured afterwards on the OVERRIDE phrasing subset for the analysis in §3; it was not used for selection. We will clarify this in Appendix F.

Retention data split. The transcription retention data use LibriSpeech dev-clean (2,703 utterances), while WER is evaluated on test-clean (2,620 utterances). These splits have disjoint speakers. The reference to “training-split utterances” in Appendix F is a typo, which we will correct.

We will add the dataset sources, splits, and overlap checks to Appendix F.

Weakness 2 & Question 2 - SFT Baseline Comparability Thank you for raising this concern. The submitted Qwen3 SFT results were obtained after half an epoch of training. We have now completed a full-epoch SFT run and evaluated OPSD-text at half an epoch for comparison. Neither method used checkpoint selection; all results are from the final checkpoint of each run. The results below use the corrected Qwen3-Omni evaluation described in our general response.

Qwen3-Omni-30B-A3BepochsIaAcc|IaReflexJointText ctrl
Base–77.164.532.849.786.3
SFT0.571.876.721.755.087.2
SFT1.075.179.718.659.894.0
OPSD-audio1.078.876.022.059.991.2
OPSD-text0.579.277.320.361.291.8
OPSD-text1.078.982.116.264.896.3

OPSD-text consistently outperforms SFT at the same number of epochs, with joint accuracy improvements of 6.2 points at 0.5 epoch and 5.0 points at 1 epoch. Moreover, SFT reduces affirmative accuracy relative to the base model, while OPSD-text preserves it. Both methods use the same LoRA configuration (rank 64, α = 128) and learning rate (1e-4). Neither method used validation-based checkpoint selection. An additional SFT run with a lower learning rate (2e-5, 0.5 epoch) achieved 55.8% Joint accuracy, compared with 55.0% at 1e-4. We will update Table 2 with the full-epoch SFT results and clarify the training settings in Appendix F.


Weakness 1 & Question 4 — Additional Baseline Comparisons

Thank you for the suggestions. We conducted all three additional comparisons on Qwen2.5-Omni-7B using the full benchmark.

Offline distillation with the same teacher. We generated offline answers using the same frozen text-conditioned teacher as OPSD-text and trained the student with the SFT recipe, keeping the backbone, training examples, and trainable parameters fixed.

SupervisionAcc|IaReflexJoint
Gold labels (SFT)75.522.852.9
Offline teacher75.321.954.8
On-policy teacher (OPSD-text)83.614.560.9

The offline teacher agrees with the gold labels on 98.1% of training examples, and offline distillation performs similarly to SFT. In contrast, OPSD-text improves Joint accuracy by 6.1 points over this offline baseline, supporting the benefit of on-policy distillation.

Two-stage inference. We implemented a two-stage pipeline: Stage 1 predicts the attribute from audio using the affirmative question, and Stage 2 executes the negated instruction in a separate text-only call using the predicted attribute.

MethodCallsAcc|IaReflexJointAcc (all items)
Base, single-call163.731.145.459.2
Base, two-stage291.55.065.267.7
OPSD-text, single-call (Seed 1)183.514.460.971.8
OPSD-text, two-stage297.11.570.771.8

Two-stage inference is a strong alternative, improving the base model’s Acc|Ia from 63.7% to 91.5% and Joint accuracy from 45.4% to 65.2%. Combining it with OPSD-text further improves Acc|Ia to 97.1% and Joint to 70.7%. However, errors in the first-stage attribute prediction propagate to the second stage, leaving the overall accuracy of OPSD-text unchanged at 71.8% despite the improvement in conditional accuracy. Moreover, two-stage inference requires explicitly decomposing the task and making two model calls, whereas OPSD-text improves single-call performance without changing the inference pipeline.

Self-refinement. We evaluated one iteration of self-refinement (answer, critique, refine), following Madaan et al., with audio provided in all three calls.

MethodCallsAcc|IaReflexJoint
Direct prompting163.731.145.4
Self-refinement343.050.930.6
OPSD-text183.614.560.9

Self-refinement substantially reduces accuracy. We observed that the critique stage rejects the initial answer approximately 90% of the time, regardless of its correctness, often causing the model to return the audio-supported attribute instead of following the negated instruction.

We will include these comparisons in Section 4.

We sincerely thank the reviewer again for the thorough and constructive feedback. Your suggestions have been particularly helpful in strengthening our evaluation.

We would like to kindly follow up on our rebuttal. We have conducted the additional experiments you suggested, including offline distillation, two-stage inference, self-refinement, and non-negation remapping, as well as clarifying the training–evaluation separation and SFT baseline comparability.

We would appreciate hearing whether these results have addressed your main concerns. If so, we would be grateful if you could consider revisiting your score.

Reviewer_dA3pOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the so-called "perception-override reflex" problem, where speech language models may fail to answer questions about audio with negated instructions. It also introduces OVERRIDE, a controlled benchmark built from MMSU, to evaluate six models. To mitigate the perception-override reflex issue, the authors use on-policy self-distillation, where the frozen model itself serves as the teacher under a privileged condition: the relevant attribute is provided in text, and the teacher is queried using a reliably executable canonical phrasing. The experiments show improvements in accuracy over the baselines.

Critical strengths
  1. The research problem deserves further investigation to some extent.
  2. The proposed OVERRIDE benchmark is useful for probing the vulnerability of pretrained speech language models, particularly to the "perception-override reflex" problem.
  3. The paper finds that reliable text-conditioned behavior provides a more suitable training signal for mitigating the reflex issue.
Critical weaknesses

Serious presentation issues:

I spent hours reading this paper and found that its writing exhibits strong signs of AI-generated content, even not polished by humans.

The wording is hard to follow. There are many strange, vague, unexplained phrases and dangling sentences in the paper, e.g., "susceptibility under audio" (line 83), "the broader phrasing effect" (line 86), "this relationship and residual variation" (line 196), and so on.

To my surprise, there is only one citation in the Introduction. I don't believe this is a paper written by human authors.

Technical concerns:

I have several major concerns:

  1. The position and novelty of this paper are unclear, as the underlying problem has already been investigated in prior work (see references below). The paper largely reframes a known issue and proposes a straightforward solution. Merely renaming an existing problem does not constitute a sufficient research contribution.
  2. The proposed method essentially combines on-policy self-distillation with text conditioning to improve pretrained models. This appears to be straightforward fine-tuning with additional data rather than a substantial technical contribution.
  3. The evaluation is inadequate. Only two relatively weak baselines are included in the main experiments, with no comparisons against prior methods.

References:

  • Yoo, HaeJun, Yongseop Shin, Insung Lee, Myoung-Wan Koo, and Du-Seong Chang. "Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval." In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 22664-22680. 2026.
  • Xu, Jingqi. "Omni-NegCLIP: Enhancing CLIP with Front-Layer Contrastive Fine-Tuning for Comprehensive Negation Understanding." arXiv preprint arXiv:2603.29258 (2026).
  • Li, Haitao, Tian Tan, Yuguang Yang, Shan Yang, and Xie Chen. "AnyAudio-Judge: A Dynamic Rubric-Based Benchmark and Evaluator for Audio Instruction Following." arXiv preprint arXiv:2606.03116 (2026).
Questions for authors

None. See weaknesses.

4 comments

Weakness1 - Presentations. Thank you for pointing out the unclear wording. We would like to clarify that we only use AI to polish the article, but we do agree that some expressions could be more direct and precise. We will revise the paragraph in Lines 82–88 as follows:

We further investigate why some negated instructions trigger the reflex more often than others (§3). Across 56 phrasings and six models, we find that instructions that are harder to follow in text also lead to more reflex errors under audio (Pearson correlations: −0.44 to −0.81). However, text-side difficulty does not explain all differences across phrasings. For example, instructions that ask models to imagine a modified version of the audio produce substantially more reflex errors than direct negations, even when their text-side accuracy is similar. These findings identify text-side instruction following as a useful predictor of the reflex and suggest that using more reliably understood instructions could help reduce it.

Similarly, we will replace the unclear expression in Line 196 with: “Finally, we show that text-side accuracy cannot fully explain the reflex: models still make errors on audio even when they can answer the same questions correctly in text (§3.3).” We will also review the manuscript for similar wording issues.

Regarding citations, the current Related Work section (§5) covers speech-LLM benchmarks, negation in language models, modality conflicts, and on-policy distillation. We agree that this discussion should appear earlier. We will move Related Work to §2 and add relevant citations directly to the Introduction to better position our contributions.


Weakness 2 - Contributions. Thank you for the comment. We agree that negation failures in multimodal models have been studied in prior work. However, the cited papers investigate different problems.

  • Omni-Embed-Audio focuses on audio-text retrieval with negative queries,
  • Omni-NegCLIP studies negation understanding in vision-language models, and AnyAudio-Judge evaluates instruction following in audio generation.

In contrast, our work investigates a specific failure in speech LLMs: models can correctly recognize an attribute from audio and follow a negated instruction when the attribute is given in text, but often fail when they must combine audio perception with the same instruction.

Our contribution goes beyond naming this failure. Specifically:

  1. Controlled evaluation. We introduce OVERRIDE with matched affirmative, negated, and text-control instructions to distinguish this failure from audio perception errors and general negation failures. We evaluate 4,204 audio clips across 40 tasks and seven speech LLMs, including proprietary models.
  2. Systematic analysis. We investigate why some instructions cause more errors than others. Across 56 phrasings and six models, we find that text-side instruction accuracy predicts the frequency of errors under audio. We further identify additional errors associated with counterfactual instructions that cannot be fully explained by text-side accuracy.
  3. Analysis-guided mitigation. Based on these findings, we use the model’s more reliable text-conditioned behavior to guide its audio-conditioned instruction following, substantially improving performance without degrading its original audio capabilities.

Clarification of scope. We would also like to clarify that our study is not limited to instructions containing explicit negation words such as “NOT.” As shown in Table 7, our 56 instruction phrasings already cover various forms of semantic negation, including asking models to identify the wrong option, reject a false claim, or select the alternative to the perceived attribute. The perception-override reflex persists across these different formulations. Therefore, our findings concern semantic negation rather than sensitivity to specific negation words.

We will clarify these distinctions from prior work and make our contributions more explicit in the revised Introduction and Related Work.

Weakness 3 - Method. Thank you for the comment. We would like to clarify that our contribution is not a new on-policy distillation algorithm, but an analysis-guided teacher design. Based on our findings in §3, we use the frozen model itself as the teacher, provide the correct audio attribute in text, and select an instruction phrasing that the teacher can reliably follow.

Regarding the method, our ablations show that the improvement is not simply due to additional fine-tuning. As shown in Table 2, on Qwen2.5-Omni, our method achieves 83.6% accuracy, compared with 75.5% for SFT, 75.7% for audio-conditioned distillation, and 78.3% for text-conditioned distillation without canonical phrasing. Furthermore, Table 3 shows that changing only the teacher’s instruction phrasing can substantially affect student performance, even when the training method remains unchanged.

We also conducted an additional comparison using offline responses from the same text-conditioned teacher. OPSD-text achieves 60.9% Joint accuracy, compared with 54.8% for offline distillation and 52.9% for gold-label SFT. This further supports the benefit of on-policy training beyond simply using a text-conditioned teacher.

Regarding the evaluation, our experiments cover both the failure itself and the proposed mitigation. Table 1 evaluates seven speech LLMs, including three proprietary models, to establish the generality of the failure. Table 2 compares our method against prompting, SFT, and different distillation variants, while Table 3 further examines the effect of teacher instruction quality. The cited prior methods address different tasks and are not directly applicable to our setting. We will clarify the scope of these comparisons in the revision.

Thank the authors for the response. I appreciate the efforts to improve the presentation and provide additional experimental comparisons.

However, my major concerns remain:

  1. Limited novelty: The identified phenomenon is interesting, but its distinction from existing negation failures and multimodal instruction-following issues remains insufficiently convincing. The proposed method also relies on standard on-policy self-distillation, with the main innovation being the use of text-privileged canonical teacher instructions.

  2. Limited generalization: The evaluation primarily relies on binary questions derived from MMSU. The four-option experiments cover only a limited subset, and the mitigation is validated on only two Qwen model families. Its effectiveness on other architectures and more realistic, open-ended tasks remains unclear.

Overall, despite the improvements, I do not believe the revisions sufficiently address my fundamental concerns regarding novelty and generalizability. I therefore maintain my original rating.

Thank you for your follow-up and for acknowledging our additional efforts.

We would like to briefly reiterate that our paper is primarily an empirical study rather than a method paper. Our main contributions are the controlled identification of the perception-override reflex and its systematic analysis across seven speech LLMs, 40 tasks, and 56 instruction phrasings. The OPSD-based approach is presented as a mitigation motivated by these findings, rather than as a novel optimization algorithm.

While prior work has investigated negation and multimodal instruction following, our study specifically characterizes the audio–text behavioral discrepancy through matched affirmative, negated, and text controls, and identifies phrasing-level patterns not established by the cited works.

We recognize that our controlled evaluation and mitigation have defined scopes, which are explicitly discussed in the paper. We hope the empirical contributions can be assessed in light of their intended research objectives.

Thank you again for your time and consideration.

Reviewer_CkP7OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies a failure of speech language models under negated instructions: models may correctly identify an attribute in audio but still return that attribute when instructed to select the alternative. The authors introduce OVERRIDE, comprising 4,204 items across 40 task tiers, with matched affirmative, negated, and text-informed conditions. Experiments across seven models establish the phenomenon, while analyses across 56 phrasings identify text-side instruction executability as a useful but incomplete predictor. The paper then uses a text-privileged, canonical teacher for on-policy self-distillation. On Qwen2.5-Omni, pooled conditional negated accuracy increases from 63.7% to 83.6%, with improvements also reported on Qwen3-Omni.

Critical strengths
  1. The controlled evaluation is the strongest contribution. Conditioning on affirmative-correct items, adding matched text controls, and reporting unfiltered joint accuracy help distinguish negated-instruction failures from ordinary task errors. The cross-model and cross-phrasing evaluations make the phenomenon substantially more convincing than isolated examples.

  2. The mitigation follows naturally from the behavioral analysis. Teacher-conditioning and teacher-phrasing ablations connect the observed text-side advantage to an effective intervention. The paper also appropriately distinguishes behavioral evidence from a causal account of the underlying mechanism.

Critical weaknesses
  1. The mitigation experiments need clearer documentation of training–evaluation separation and baseline comparability. Sections 3–4 establish that evaluation phrasings are excluded from training, but Appendix F does not clearly establish disjointness of underlying audio identities. In addition, Qwen3-Omni distillation uses one epoch while SFT uses half an epoch. These details affect how strongly the results establish generalization and superiority over SFT. I am identifying missing clarification, rather than asserting that overlap occurred.

  2. The main setting is a controlled two-option transformation. Appendix A usefully extends the analysis to four options, but broader applicability to naturally occurring, open-ended negated requests remains uncertain. Supplying the true attribute explicitly in the text control also creates an information advantage; consequently, the evidence supports a behavioral audio–text discrepancy more directly than a specific mechanism of perceptual dominance.

Questions for authors
  1. Are the underlying audio clips and source identities used for override, anchor, and retention training disjoint from OVERRIDE and the capability-retention evaluations? Please provide the source datasets, split rules, and overlap counts.

  2. Why was Qwen3-Omni SFT trained for half an epoch while the distilled variants were trained for one epoch? Does the advantage persist with matched data exposure and independently selected SFT hyperparameters?

3 comments

We sincerely thank the reviewer for their positive assessment of our work and constructive feedback. We appreciate their recognition of our contributions.


Weakness 1 & Question 1 - Training / evaluation separation. Yes, they are disjoint. All training audio (override, anchor and retention groups; 16,590 clips) comes from LibriTTS-R, CREMA-D, LibriSpeech dev-clean, AudioSet, AudioSet strong-label, TACOS, MagnaTagATune, Clotho v2 and MusicCaps, with the labels these corpora provide. All evaluation audio comes from elsewhere: OVERRIDE (4,204 clips) and the four-choice retention set (4,981 clips) are MMSU, built from MELD, GigaSpeech, Common Voice, Emilia, CoVoST 2, EDACC, VCTK, CHILDES, SLURP, SEAME, Fake-or-Real, RAVDESS and Switchboard; WER uses LibriSpeech test-clean. We double-checked the source of every training clip, none of them appear in any evaluation set.


Weakness 1 & Question 2 - Qwen3 SFT epochs. The half-epoch SFT checkpoint was the one available when the tables were frozen. We have since trained SFT for a full epoch and evaluated the half-epoch OPSD-text checkpoint. All numbers use the corrected Qwen3-Omni evaluation described in the general response. Pooled negated block as in Table 2:

Qwen3-Omni-30B-A3BepochsIaAcc|IaReflexJointText ctrl
Base–77.164.532.849.786.3
SFT0.571.876.721.755.087.2
SFT1.075.179.718.659.894.0
OPSD-audio1.078.876.022.059.991.2
OPSD-text0.579.277.320.361.291.8
OPSD-text1.078.982.116.264.896.3

At matched exposure OPSD-text leads SFT by 5.0 joint points at one epoch and 6.2 at half an epoch; the difference comes from the affirmative items, where SFT falls below the base model and OPSD-text does not.

Neither method was tuned beyond the standard LoRA setting for this model; the one additional SFT learning rate we tried (2e-5) changes joint accuracy by less than one point. The remaining difference is the effective batch (SFT 32 on 4 GPUs, distillation 64 on 8 GPUs), now stated in Appendix F. The revised Table 2 uses the full-epoch SFT row.


Weakness 2 - Scope of the two-option setting and the text control

Thank you for raising this. Following the suggestion, we tested the open-ended form: on the five attributes whose values are category labels (accent, emotion, gender, speaker count, dialogue turns; all 570 such items; Qwen2.5-Omni) we asked "What is the {attribute}?" and "Which value is NOT the {attribute} here?" with no options. Numbers are reflex rates (%, lower is better): on items where the open affirmative question names the true attribute (n), the share of open negated questions still answered with that attribute.

attributen (base / OPSD-text)basebase, text controlOPSD-text
accent7 / 1242.90.08.3
dialogue turns18 / 2022.20.020.0
emotion88 / 951.10.00.0
gender116 / 11713.80.03.4
speaker count25 / 2460.00.00.0
all254 / 26815.40.03.4

Without options the reflex is still there (15.4%), the text control is at 0%, and OPSD-text brings it down to 3.4%. The four word-level attributes (elongated word, stressed word, homophone, non-verbal sound) are excluded because the open negated question is ill-posed for them: any other word is a correct answer, and even in the text control, where the prompt states the word, the model answers with that same word on 56–66% of items, so the audio numbers would not measure the reflex.

Text control. We agree that providing the ground-truth attribute in text changes how the information is presented. Our goal is not to identify the mechanism of perceptual dominance, but to test whether the model can follow the same negated instruction when the attribute is provided in text. Together with the affirmative-correct subset, this allows us to identify a behavioral failure that cannot be explained by affirmative-task errors or text-side instruction failures alone. We will clarify this distinction in the paper.

Thank you for the responses. I have also carefully read the comments of the other reviewers. The clarification of training–evaluation separation and the matched-epoch SFT comparisons substantially address my main concerns. Meanwhile, the advantages over two-stage inference depend on the evaluation metric, and generalization to natural, open-ended requests remains insufficiently established. Overall, I continue to view the empirical contribution positively and will maintain my original score of weak acceptance.

We sincerely thank the reviewer for their thoughtful assessment and continued support of our work. We appreciate your acknowledgment that the additional experiments and clarifications have addressed the main concerns. We will also carefully consider the remaining limitations and clarify them in the final revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.