acceptodds
Under review as a conference paper at ICLR 2027

Do Omni Models Use How We Speak? Diagnosing and Improving Paralinguistic Understanding

Abstract

Strong performance on speech and multimodal benchmarks does not establish whether omni models use how an utterance is spoken to guide their decisions. We investigate this question through controlled translation selection, asking models to choose the translation better synchronized with the audio while varying vocal expression, holding the transcript and candidate translations fixed, and testing both candidate orders. Evaluated models, including commercial closed-source models, often fail to adjust their choices correctly as vocal expression changes, and most choices persist even when speech is replaced with silence, indicating limited use of vocal information beyond the transcript. We introduce Paralinguistic Counterfactual Training (PCT), which combines same-transcript audio pairs with opposite target choices and human translation-quality preferences, using candidate-order augmentation and supervised low-rank adaptation. On Qwen2.5-Omni-3B and 7B, PCT increases paired correctness from 2.65% to 49.29% and from 8.17% to 50.71%, respectively, with gains also observed in four further post-trained omni models. Additional evaluations show transfer to general audio understanding and paralinguistic tasks, and largely preserved performance on other evaluated input modalities for Qwen2.5-Omni. These findings show that explicitly linking vocal expression to correct decisions can improve audio use beyond the supervision task.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Omni Models, Audio Understanding, Paralinguistic Understanding, Multimodal Alignment

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_RvWROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether omni models use paralinguistic cues, such as prosody and emotion, when making decisions. The authors introduce a controlled translation-selection task in which recordings with identical transcripts but different vocal expressions support different translations. They find that several omni models rarely make the correct audio-dependent choice. To address this, they propose Paralinguistic Counterfactual Training, which combines counterfactual audio supervision with human translation-quality preferences. PCT substantially improves paired correctness and shows transfer to general audio understanding and paralinguistic tasks.

Critical strengths
  1. A well-motivated diagnostic. The controlled evaluation distinguishes general translation-quality judgment from the ability to use vocal information when it changes the correct answer. The silence control provides additional evidence of this gap.

  2. A simple and effective intervention. PCT uses standard supervised fine-tuning with LoRA. The quality-only and counterfactual-only ablations support combining both supervision sources.

  3. Broad evaluation. Experiments cover multiple model families, audio understanding benchmarks, paralinguistic tasks, and cross-modal capability retention.

Critical weaknesses
  1. The contribution of mixed supervision is not fully isolated. The paper does not specify whether the single-source controls match PCT in training steps and example exposure, and Counter-only results on the main DF metric are missing. Numerical training settings, including the learning rate and LoRA configuration, are also needed to assess and reproduce these comparisons.

  2. DF provides evidence of audio-dependent decisions, but leaves candidate-order robustness unclear. It averages paired correctness separately over the two candidate orders rather than requiring all four views to be correct. Reporting order consistency alongside DF would clarify whether the improved choices reliably survive candidate reversal.

Questions for authors
  1. Could you report candidate-order consistency, all-four-view correctness, and TIE/invalid-output rates before and after PCT?

  2. What DF does Counter-only achieve? Were the single-source controls matched to PCT in training steps and example exposure? Please also provide the numerical training hyperparameters and checkpoint-selection criteria.

  3. Can the transfer results be validated on an independent benchmark involving natural conversational speech or unseen paralinguistic phenomena? For cross-modal retention, could you also provide uncertainty estimates or formal non-inferiority tests?

4 comments

Thank you for your insightful comment.

Response to Weakness 1

We provide additional details on the training budgets, performance comparisons, and numerical configurations of single-source and mixed supervision.

1. Training Budget

All supervision configurations use identical training budgets and optimization settings.

ConfigurationPCTQuality-onlyCounter-only
Training rows10,71210,71210,712
Optimizer updates1,3391,3391,339
Rows per update888
Learning rate1.00E-041.00E-041.00E-04
LR schedulerCosineCosineCosine

2. Counter-only vs. Mixed PCT

ModelSupervisionDF (%)IWSLT HA (%)
Omni-3BCounter-only39.3666.89
Omni-3BMixed PCT49.2971.99
Omni-7BCounter-only41.2767.35
Omni-7BMixed PCT50.7174.73

Compared with Counter-only, Mixed PCT improves DF by 9.93 pp and 9.44 pp on Omni-3B and Omni-7B, respectively. It also improves IWSLT HA by 5.10 pp and 7.38 pp, respectively.

These results demonstrate that combining counterfactual audio supervision with human translation-quality preferences improves both audio-conditioned translation decisions and general translation quality.

3. Numerical Training Configuration

All training experiments use the same LoRA and optimization configurations.

ParameterConfiguration
LoRA rank16
LoRA alpha32
LoRA dropout0
Target modulesq_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Adapted layersAll Thinker LLM layers
OptimizerAdamW
Learning rate1.00E-04
SchedulerCosine
Warmup steps134
Gradient clipping1
PrecisionBF16
Training updates1,339
Checkpoint selectionTerminal checkpoint (adapter_last)

Response to Weakness 2 and Question 1

We provide additional analyses of candidate-order consistency and all-four-view accuracy to further evaluate the stability of PCT's decisions under candidate-order reversal.

1. Candidate-order Consistency (Omni-7B)

MetricBasePCT
DF8.17%50.71%
Order Consistency54.07%66.78%
All-four-view Accuracy2.12%42.40%
TIE250
Invalid00

Order consistency is calculated based on translation identity across AB/BA candidate orders, rather than directly comparing output letters.

2. All-four-view Accuracy (Omni-3B)

MetricBasePCT
DF1.59%49.29%
All-four-view Accuracy0.00%40.99%
  • For Omni-7B, PCT improves candidate-order consistency from 54.07% to 66.78% and all-four-view accuracy from 2.12% to 42.40%.
  • For Omni-3B, PCT similarly increases all-four-view accuracy from 0% to 40.99%.

These results demonstrate that PCT improves not only audio-conditioned translation decisions but also the consistency of these decisions under candidate-order reversal.

3. TIE and Invalid Outputs

For Omni-7B, Base produces 25 TIE outputs, whereas PCT produces neither TIE nor invalid outputs. Neither configuration produces invalid outputs.

Response to Question 2

  • Counter-only Performance: Counter-only achieves DF scores of 39.36% and 41.27% on Omni-3B and Omni-7B, respectively. PCT combines two equally weighted supervision sources: translation-quality supervision (DQD_Q) from IWSLT and counterfactual audio supervision (DCD_C) from ContraProST. DQD_Q promotes translation-quality judgments, while DCD_C teaches the model to select translations according to vocal conditions. Their combination achieves higher DF scores of 49.29% and 50.71%, respectively, while also improving IWSLT human agreement. See the performance comparison in our response to Weakness 1.

  • Training Budget: Quality-only, Counter-only, and Mixed PCT use the same total training exposure of 10,712 presentations, 1,339 optimizer updates, and 8 rows per update, with the same learning rate 1e−41e-4 and cosine schedule. See the training-budget comparison in our response to Weakness 1.

  • Training Hyperparameters: All experiments use the same LoRA and optimization configurations, including rank 16, alpha 32, dropout 0, AdamW, and BF16. The complete configuration is provided in our response to Weakness 1.

  • Checkpoint Selection: All experiments use the terminal checkpoint (adapter_last) after 1,339 optimizer updates, without dev-set-based checkpoint selection.

Response to Question 3

We provide additional evaluations on independent real speech and cross-modal retention to examine PCT's transferability and its impact on other modalities.

1. Independent Real-speech Evaluation

We evaluated PCT on the independent StressTest benchmark using real recorded speech, assessing stressed-word detection and stress interpretation.

TaskModelBasePCTImprovement95% CI
Stressed-word DetectionOmni-3B21.80%52.90%+31.2 pp[24.8, 37.7]
Stress InterpretationOmni-3B40.40%63.30%+22.9 pp[16.4, 29.6]
Stressed-word DetectionOmni-7B15.30%54.10%+38.8 pp[31.1, 46.7]
Stress InterpretationOmni-7B54.60%63.80%+9.2 pp[3.7, 14.8]

All four improvements have 95% bootstrap confidence intervals entirely above zero, demonstrating that PCT's improvements transfer to an independent real-speech evaluation.

2. Cross-modal Retention

We further evaluated whether audio-focused post-training affects the model's existing capabilities in other modalities, including text, video, vision, and audio-visual understanding.

We performed paired evaluations on identical test examples and calculated 95% confidence intervals using 10,000 paired bootstrap samples.

ModelBenchmarkBasePCTΔ (pp)95% CI
3BMMLU-Redux55.7756.690.92[−0.22, +2.09]
3BMLVU63.8062.78−1.01[−2.02, −0.05]
3BMMStar54.6054.870.27[−1.07, +1.67]
3BDailyOmni58.1957.19−1.00[−2.46, +0.44]
7BMMLU-Redux66.5766.39−0.18[−1.03, +0.66]
7BMLVU65.9666.560.60[−0.22, +1.43]
7BMMStar59.4060.601.20[−0.13, +2.54]
7BDailyOmni61.5461.710.17[−1.18, +1.48]
  • Omni-7B: None of the four cross-modal performance differences is statistically significant.
  • Omni-3B: Three of the four differences are not statistically significant, while MLVU shows a small decrease of 1.01 pp.

Overall, these results suggest that PCT's audio-focused post-training largely preserves the model's existing capabilities across other modalities, without introducing consistent cross-modal degradation.

Dear Reviewer RvWR,

We sincerely appreciate your time and effort in reviewing our manuscript and offering valuable suggestions.

As the author-reviewer discussion phase is drawing to a close, we would like to confirm whether our responses have effectively addressed your concerns.

We provided detailed responses to your concerns a few days ago, and we hope they have adequately addressed your issues. lf you require further clarification or have any additional concerns, please do not hesitate to contact us. We are more than willing to continue our communication with you.

Thank you for the detailed responses and additional analyses. They address my main concerns about the training setup, candidate-order robustness, and transfer to independent real-speech data. Overall, I maintain my original rating 3: Weak acceptance.

Reviewer_m4MJOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper evaluates whether speech Omni models (e.g. Qwen2.5-Omni) are able to make decisions based on how something is said, rather than just what is said, and proposes a LoRA finetuning approach to improve performance. As their main training and evaluation dataset, the authors reuse ContraProST, an existing synthetic speech translation benchmark. Each example in it has two recordings, spoken differently of the same English sentence, that would call for different translations. The model hears one recording at a time, plus the transcript, and picks the better of two translations. The example counts as correct only if the model gets both recordings right (the paper calls this Directed Flip, DF). Open and commercial models score at or below chance on DF and most of their answers stay the same even when the audio is replaced by silence. The proposed approach, PCT, is LoRA fine-tuning on ContraProST's training split mixed with human translation quality preferences. On Qwen2.5-Omni models, PCT improves DF from under 10% to about 50%. The paper also reports gains on general audio and emotion benchmarks for some models and drops for others.

Critical strengths
  1. The initial evals give some useful information. Commercial models as well as models post-trained with RL for speech reasoning all fail the ContraProST DF eval metric, and training only on human quality preferences makes a model rely on the transcript even more. Showing this is useful because it means the problem is still open for current models, though the eval itself comes from ContraProST, a pre-existing benchmark, and is not created by the authors.
  2. PCT is straightforwad and its in-domain gains are large and consistent across both Qwen2.5-Omni models and four post-trained variants. The paper also checks carefully that performance on text, image and video tasks barely changes. Thus the paper shows the performance can be improved with simple finetuning, though I am not convinced the gains carry beyond this benchmark (Critical Weakness 2).
Critical weaknesses
  1. The paper does not present a novel eval or training approach. At its core, the gains seem to largely be due to fine-tuning on the eval benchmark's own training split.
    1. The gap between recognizing and acting on vocal cues is well known. StyleTalk (cited by paper), already trains on same-sentence pairs that need different responses. EchoMind (https://arxiv.org/abs/2510.22758) and StressTest (https://arxiv.org/abs/2505.22765) show that speech LMs often fail to act on vocal cues. Even specifically for the translation task, the ContraProST paper (used by this paper for training and eval) already showed that ST models rarely get both recordings right, and that this is only weakly correlated to general translation performance. Züfle et al., also cited, already showed that finetuning an Omni model on human quality scores makes it worse on ContraProST, which is the Quality-only experiment in this paper.
    2. DF is the "global" metric defined in the ContraProST paper, just recast as a multiple-choice question. Swapping candidate order to avoid position bias and requiring every item in a contrastive pair to be correct to get points is standard practice.
    3. The fix is not novel either. Training on pairs with the same text but different answers, so that a model cannot ignore the other modality, is done by StressTest. PLaST (https://ojs.aaai.org/index.php/AAAI/article/view/40449) trains paralinguistic-aware translation and improves on ContraProST.
    4. Other work that is worth citing: SD-Eval (https://arxiv.org/abs/2406.13340), CP-Bench (https://arxiv.org/abs/2509.16589), ParaS2S (https://arxiv.org/abs/2511.08723) and the concurrent ParaBridge (https://arxiv.org/abs/2606.10581).
  2. The experiments mainly show that PCT improves in-domain performance; I am not convinced that it generalizes beyond the ContraProST benchmark.
    1. Training and evaluation use the same benchmark, the same prompt and likely the same TTS voices (unless clarified to not be), since the split is by sentence. Real speech or unseen voices are not evaluated on, though this is possible (experiment request 1). The emotion results use real speech, but they test emotion recognition (which the paper itself says is not the bottleneck), rather than the hearing-acting gap.
    2. The general-audio results could be affected by answer-parsing failures. Some baselines get results far below chance (e.g. 3% on a four-option questioon) or far below their own published numbers, which is unusual. Since PCT trains the right answer format, some of the transfer could just be better formatting!
    3. Table 1 should be better presented as a regression check rather than evidence of external transfer; why would a change of about 2 points be trustworthy here? At least variance between training runs should be reported, given that ablations shows swings of 5 to 6 points between training variants.
  3. The extra analyses in Section 5 don't really support what they claim. This matters because they are the evidence that PCT changes how the model processes vocal information.
    1. The layer analysis rescales each layer's curves to max out at 1, which hides how much each layer actually changed. If PCT simply made the deeper layers have a larger magnitude of response, the early layers would look flat after rescaling even if they did not change at all. Note that larger magnitude across the board does not imply a shift of prosody-specific sensitivity.
    2. The noise analysis shows that PCT's emotion-classification improvements survive background noise. But this says nothing about whether its translation performance stays robust. It also shows only the 3B model.
    3. As a smaller comment, key training details such as the LoRA settings and the promised full fine-tuning comparison are missing.
Questions for authors

Here are some clarification questions:

  1. The paper uses about half of ContraProST's examples, why only half? Do test speakers appear in training?
  2. How many parsing failures are there for each model/benchmark? Why are several baselines far below their published scores?
  3. What is Open-R1-GRPO-3B?

and here are some experiment requests:

  1. Evals on unseen voices or real speech, e.g. held-out TTS voices. Some held out benchmark results could be StressTest's real recordings, or MuST-SHE (real speech where the right translation depends on the speaker's voice).
  2. A no-audio control (PCT trained with silence instead of audio).
9 comments

Thank you for your insightful comment.

Response to Weakness 1-1

Prior work has established the gap between recognizing vocal cues and acting on them. Our work investigates how to improve audio-conditioned translation decisions in omni models through counterfactual supervision.

Main Results

EvaluationMetricOmni-3B BaseOmni-3B PCTOmni-7B BaseOmni-7B PCT
Translation selectionDF (%)2.6549.298.1750.71
Stress recognitionAccuracy (%)2442——
Meaning interpretationDF (%)1.338.6——

Supervision Ablations

Training ConfigurationOmni-3B DF (%)Omni-7B DF (%)
Base2.658.17
Quality-only1.592.12
PCT-Unpaired26.33—
PCT-NoAudio1.06—
PCT49.2950.71

PCT-Unpaired underperforms the matched-budget PCT control by 21.02 pp (95% CI: [15.08, 27.35]). PCT-NoAudio underperforms PCT by 46.29 pp (95% CI: [38.89, 53.55]).

Differences from Prior Work

  • StyleTalk / Spoken-LLM: Studies spoken-dialogue response generation under different speaking styles with identical lexical content. We study translation selection, where the model must choose between fixed candidate translations according to contrasting vocal expressions.
  • StressTest: Investigates how sentence stress affects semantic understanding and model behavior. We formulate this problem as audio-conditioned translation selection and combine counterfactual audio supervision with human translation-quality preferences.
  • EchoMind: Provides multi-level evaluations of speech understanding, reasoning, and response generation. Our focus is on improving the influence of acoustic evidence on translation decisions through controlled counterfactual training.
  • ContraProST: Reveals that speech translation systems often fail to correctly handle recordings with contrasting prosody. We extend this investigation to omni models and study a corresponding post-training intervention.
  • Züfle et al.: Their findings are consistent with our Quality-only results: improving general translation-quality preferences does not necessarily improve speech-specific translation judgments.

Our contribution is therefore not the introduction of the recognition–action gap itself, but a counterfactual supervision approach that explicitly connects contrasting acoustic evidence to translation decisions. The ablations show that neither quality-preference supervision alone nor training without real contrasting audio achieves the gains of full PCT.

Response to Weakness 1-2

DF builds upon the Global Condition CGC_G introduced in ContraProST and adapts it to generative candidate translation selection in omni models. Section 2 presents details of DF.

Specifically:

  1. Audio contrast: The transcript and candidate translations are fixed, while the model must select the correct translation for each of the two contrasting audio conditions.
  2. Candidate-order control: Both AB and BA candidate orders are evaluated to control for positional bias.
  3. Joint correctness: A sample is considered successful only when both contrasting audio conditions receive the correct translation selection.

This design directly measures whether an omni model adjusts its translation decisions according to acoustic variations, providing a corresponding evaluation criterion for assessing PCT.

Response to Weakness 1-3

Prior work has explored the use of contrasting speech conditions to improve paralinguistic understanding. StressTest constructs training examples with identical text but different stress patterns and corresponding interpretations, while PLaST introduces a dual-branch architecture to model linguistic and paralinguistic information. We extend this direction to existing omni models through post-training, combining counterfactual audio supervision with translation preferences without modifying the model architecture.

We evaluate PCT at three levels:

EvaluationMetricOmni-3B BaseOmni-3B PCTOmni-7B BaseOmni-7B PCT
Stress recognitionAccuracy (%)24422058
Meaning interpretationDF (%)1.338.64.239.9
Translation selectionDF (%)2.6549.298.1750.71

On real recorded speech from StressTest, PCT further improves stressed-word detection by 31.2 pp and 38.8 pp for Omni-3B and Omni-7B, respectively, and stress interpretation by 22.9 pp and 9.2 pp. These results use the frozen JSON-format prompt.

Two additional controls validate the role of counterfactual supervision:

Training ConfigurationOmni-3B DF (%)
Base2.65
PCT-Unpaired26.33
PCT-NoAudio1.06
PCT47.35

PCT-Unpaired underperforms PCT by 21.02 pp (95% CI: [15.08, 27.35]), while PCT-NoAudio underperforms it by 46.29 pp (95% CI: [38.89, 53.55]).

These results show that PCT improves not only vocal-cue recognition and meaning interpretation, but also their utilization in translation decisions. The ablations further demonstrate that balanced supervision over contrasting audio conditions provides substantial gains beyond single-condition training.

Response to Weakness 1-4

We further examined SD-Eval, CP-Bench, ParaS2S, and the concurrent ParaBridge, and evaluated Base and PCT models on the available independent benchmark data.

EchoMind Results

TaskOmni-3B BaseOmni-3B PCTOmni-7B BaseOmni-7B PCT
Human Understanding62.2762.1165.8163.13
Human Reasoning57.6960.2160.7363.35
TTS Understanding64.4062.2867.0262.06
TTS Reasoning56.6759.5756.5861.15

Paired bootstrap results show consistent gains on reasoning tasks: +2.52 pp (95% CI [1.02, 4.03]) and +2.62 pp (95% CI [1.19, 4.05]) on human-recorded speech for Omni-3B and Omni-7B, respectively; and +2.91 pp (95% CI [1.89, 3.94]) and +4.57 pp (95% CI [3.60, 5.54]) on TTS speech.

Evaluation Availability

  • SD-Eval: We obtained part of the dataset, but its primary evaluation relies on GPT-4o judging open-ended responses, and the target subsets lack complete audio and evaluation resources. Thus, the original evaluation protocol could not be reproduced.
  • CP-Bench: The available release has missing audio and inconsistencies in dataset size, while its official Llama-3-70B judge was unavailable. Therefore, its complete evaluation protocol could not be reproduced.
  • ParaS2S: The complete evaluation dataset was not publicly available. Its speech-to-speech generation task also differs from our text-based candidate translation selection task.
  • ParaBridge: This work proposes an on-policy self-distillation method rather than a standalone benchmark. We therefore evaluated our models on its associated EchoMind MCQ data.

Differences in Research Focus

  • SD-Eval / CP-Bench: Focus on multidimensional paralinguistic perception and contextual reasoning. We further investigate whether acoustic variations influence translation decisions between candidate translations.
  • ParaS2S: Studies content and expressive-style alignment in speech-to-speech generation. We focus on audio-conditioned decisions among fixed translation candidates.
  • ParaBridge: Uses on-policy self-distillation to improve paralinguistic perception and conversational behavior. PCT instead combines counterfactual audio supervision with translation preferences to improve translation selection in omni models.

Response to Weakness 2-1 and Experiment Request 1

We further evaluated PCT on StressTest's real recorded speech to examine its transfer to real-world vocal cues.

EvaluationOmni-3B GainOmni-7B Gain
Stressed-word detection+31.2 pp+38.8 pp
Stress interpretation+22.9 pp+9.2 pp

These results demonstrate that PCT improves both vocal-cue recognition and the interpretation of stress-induced meaning in real speech, extending beyond the synthetic speech conditions used in ContraProST.

Response to Weakness 2-2

All General Audio results were evaluated using the respective benchmarks' official Pass@1 evaluation scripts. We further audited the available raw generations to examine potential parsing failures.

For AIR-Bench Foundation, the recovered Omni-3B Base outputs achieved 67.38%, exactly reproducing the reported result, with only 5 empty outputs among 24,683 examples and no parsing failures (failure rate: 0.02%; 95% CI: [0.01, 0.05]).

The historical records also identify a format-related scoring issue for Kimi-Audio on MMSU, where bare option numbers were incompatible with the official string-matching parser. We have distinguished this issue from genuine incorrect predictions and retained the official scores.

Response to Weakness 2-3

We conducted an additional evaluation of Omni-3B on three General Audio benchmarks using the official evaluation protocols. The results are summarized below.

BenchmarkBasePCTImprovement95% Bootstrap CI
MMAU69.2072.10+2.90 pp[1.50, 4.30]
AIR-Bench67.3468.99+1.65 pp[1.29, 2.01]
MMSU60.3765.92+5.54 pp[4.62, 6.49]
Average65.6469.00+3.36 pp[2.79, 3.95]
  • PCT consistently outperforms Base across all three benchmarks.
  • All three improvements are statistically significant, with their 95% paired bootstrap confidence intervals entirely above zero.
  • The average performance improves by 3.36 percentage points (95% CI: [2.79, 3.95]).

These results further demonstrate that PCT's improvements extend beyond the ContraProST evaluation to general audio understanding benchmarks.

Response to Weakness 3-1

We re-examined the original layer-wise measurements and revised Figure 8 to use a shared absolute scale across Base and PCT.

  • At Layer 9, the prosody-contrast distances are nearly identical (0.002195 vs. 0.002164).
  • At Layer 27, the distance increases from 0.000559 (Base) to 0.035185 (PCT), approximately 63 times higher.

This shift is consistent with recent findings [1], which show that audio information primarily shapes candidate representations in early-to-middle layers, while fine-tuning strengthens the contribution of candidate information to final answer selection in middle-to-late layers.

Accordingly, we revise our interpretation of Figure 8: PCT is associated with substantially increased audio-conditioned representational differences in middle-to-late layers, consistent with a stronger role of acoustic information in subsequent decision processing.

[1] Tracing Audio Grounding and Answer Selection in Audio LLMs (2026).

Response to Weakness 3-2

We extended the MUSAN noise robustness evaluation to Omni-7B and report the results alongside the existing Omni-3B results.

Noise Robustness Results (%)

DatasetModelClean30 dB20 dB
CREMA-D3B Base6.083.802.43
3B PCT7.926.435.23
7B Base41.1232.2922.57
7B PCT43.7235.1124.18
RAVDESS3B Base7.843.151.89
3B PCT11.617.636.70
7B Base41.3323.3817.96
7B PCT45.0923.0018.56

On Omni-7B, PCT improves CREMA-D performance by 2.60, 2.82, and 1.61 pp under clean, 30 dB, and 20 dB conditions, respectively. On RAVDESS, the corresponding changes are +3.76, −0.38, and +0.60 pp.

These additional results extend the noise robustness evaluation to both model scales and show that PCT's emotion-classification gains are largely retained under background noise. We clarify that this analysis concerns emotion classification rather than translation robustness.

Reviewer_791iOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Here the authors study whether omni models use vocal cues when choosing between translations. After identifying the problem that these models do not rely on these cues, the authors then propose PCT, a supervised fine-tuning recipe using counterfactual audio examples and translation-quality preferences.

Critical strengths
  • What I particularly liked is that the authors also evaluate other audio tasks and check what happens to performance on other input modalities because this gives a real overview of the differences in a variety of tasks.
Critical weaknesses
  • W1. For me the biggest issue is that the main contribution is unclear. PCT uses standard supervised fine-tuning with LoRA on examples constructed from existing datasets. There is no additional counterfactual or consistency objective. I understand that data construction can still be a contribution, but the naming and presentation make the method sound more "special" than it actually is or than the experiments show. What is also missing is that the authors do not compare their "approach" with additional training on comparable relevant data, using the same training budget but without the balanced counterfactual construction.
  • W2. What is also missing is that the authors describe a recognition-utilization gap, but they do not show that the models recognize the relevant cues in the examples where they fail to use them. Agreement with human translation preferences on IWSLT does not establish recognition of prosody on ContraProST. The model might fail to perceive the cue, interpret it, or connect it to the translation.
  • W3. I am still unsure how well the task captures the problem the authors want to address. Stress and pauses can change meaning, but an angry delivery does not necessarily mean that a translation should use more emotionally loaded words, as suggested by Figure 1. What would be interesting is if they would report human agreement on the evaluated set or even explain how the candidate preferences were validated. It is also unclear how much the improvement in choosing between translations would carry over to generating translations.
  • W4. Can the authors check how much of the improvement comes from producing the expected answer format because they use case-sensitive exact matching without label normalization? The problem is that if a correct emotion is followed by punctuation or an explanation, it can therefore count as incorrect, while the proposed training encourages short categorical answers. So it is not clear if that is the main source of improvement.
  • W5. Several details to understand the training are missing. The appendix says that adaptation configurations, optimization schedules, resource costs, and full-fine-tuning comparisons are reported, but I could not find them. Which layers are adapted, what are the LoRA rank and scaling, and what learning rate, batch size, and training schedule are used?
  • W6. I really didn't like the +/- 0.5-point band in Figure 7. Even if it is described as serving as a visual reference, it misleads the interpretation for me because in other figures they use this for CIs.
  • W7. I also found it very confusing that the emotion labels they used were not always the same and were not consistently used across experiments without a particular argument. Consider Appendix H, which extracts the emotions from filenames and explicitly differs them from VoiceVote labels, but then Appendix F says it uses labels in an "evaluation manifest".
  • W8. There are, according to my understanding, also several inconsistencies in the reported numbers. The introduction reports real-silence agreement of 60.1-87.1%, while Section 2.3 reports 77.1-90.9%. Or the discussion of Table 2 calls 72.10 the average, although this is the MMAU score and the average is 69.00. Then also, Tables 1 and 8 give different MMAU scores again. I didn't understand if that is due to a different evaluation setup or why it's not consistent.
  • W9. I did not understand what conclusion to draw from Figure 8 beyond a change in the relative distribution of cosine distances across layers.
Questions for authors

See weaknesses.

7 comments

Thank you for your insightful comment.

Response to Weakness 1

We clarify that PCT's contribution lies in counterfactual supervision construction rather than a new optimization objective. PCT uses standard LoRA-based supervised fine-tuning (SFT) to organize contrasting audio conditions, their corresponding translation preferences, and human translation-quality preferences into a unified training dataset.

To evaluate the contribution of balanced counterfactual construction, we conduct a matched-budget PCT-Unpaired experiment:

Training ConfigurationOmni-3B DF (%)
Base2.65
PCT-Unpaired26.33
PCT47.35

PCT and PCT-Unpaired use the same total training exposure of 5,356 view presentations. PCT-Unpaired retains only one real audio condition per transcript, together with both candidate orders, whereas PCT jointly uses both contrasting audio conditions and their corresponding translation preferences.

PCT outperforms PCT-Unpaired by 21.02 pp (95% CI: [15.08, 27.35]), demonstrating the additional benefit of jointly supervising contrasting audio conditions under the same training budget.

Thus, PCT contributes a balanced counterfactual supervision construction that explicitly connects acoustic variations with translation decisions, and empirically demonstrates its advantage over single-condition training.

Response to Weakness 2

We conducted a sample-level cross-analysis of Meaning Probe and Translation Selection to examine whether models effectively use recognized vocal meaning in translation decisions.

1. Sample-level Recognition–Utilization Analysis

We aligned Meaning Probe and Translation Selection predictions using logical example IDs across the same 283 ContraProST evaluation examples. Each example was classified into four categories:

ModelMeaning ✓ Translation ✓Meaning ✓ Translation ✗Meaning ✗ Translation ✓Meaning ✗ Translation ✗
Omni-7B Base5 (0.88%)24 (4.24%)45 (7.95%)492 (86.93%)
Omni-7B PCT228 (40.28%)40 (7.07%)56 (9.89%)242 (42.76%)
Omni-3B Base0 (0%)8 (1.41%)9 (1.59%)549 (97.00%)
Omni-3B PCT210 (37.10%)58 (10.25%)58 (10.25%)240 (42.40%)

2. Recognition–Utilization Gap

Among examples where the Meaning Probe correctly identifies the intended meaning, we further measure the proportion for which Translation Selection fails:

ModelBasePCT
Omni-7B82.80%14.90%
Omni-3B100%21.60%
  • For Omni-7B, the conditional failure rate decreases by 67.83 pp (95% CI: [−84.58, −52.54]).
  • For Omni-3B, it decreases by 78.36 pp.

These results show that PCT substantially reduces the gap between recognizing vocal meaning and using it to make correct translation decisions.

3. Independent Stress Recognition

On a separate Stress Recognition evaluation, the accuracy of identifying stressed words also improves:

ModelBasePCT
Omni-7B20%58%
Omni-3B24%42%

Together, these findings provide sample-level behavioral evidence that PCT improves both vocal-cue recognition and the utilization of recognized meaning in translation decisions.

Response to Weakness 3

1. Task Validity

Our evaluation covers five categories of paralinguistic phenomena in ContraProST, rather than a single emotion-recognition or sentence-stress task. Given a fixed transcript, contrasting audio conditions convey different intended meanings and correspond to different candidate translations. Our objective is to evaluate whether models select translations consistent with the speaker's intended meaning based on vocal expressions, rather than assuming that emotional delivery must be translated using more emotionally loaded words.

2. Human Preference and Gold Provenance

We follow the official data construction and evaluation protocols of ContraProST and IWSLT, respectively:

  • ContraProST: Gold preferences are constructed based on the benchmark's intended-meaning-to-translation correspondence, following its official data quality-control procedure.
  • IWSLT: Translation preferences are derived from the official human translation-quality ratings, with higher-quality candidates receiving preference labels.

These two sources provide complementary supervision for audio-conditioned translation preferences and general translation quality.

3. Translation Selection vs. Generation

Our work focuses on audio-conditioned translation selection among given candidates, rather than open-ended translation generation. Extending PCT to open-ended speech translation generation is a direction for our future work.

Response to Weakness 4

We followed the evaluation protocols corresponding to each benchmark and further audited the parsing failures caused by output-format incompatibilities.

1. Evaluation Protocol

The label-parsing methods used for each benchmark are as follows:

  • SAVEE: Whole-word label matching.
  • CREMA-D: Case-sensitive exact matching.
  • RAVDESS: Exact matching, while some historical results used substring matching.
  • VocalSound: Whole-word label matching.

2. Formatting Failures

For CREMA-D and RAVDESS evaluations using strict exact matching, formatting-related parsing failures were negligible:

  • Omni-3B: Zero formatting-related parsing failures.
  • Omni-7B: The numbers of formatting-related failures across the four corresponding evaluation settings were 1, 4, 1, and 0, respectively.

Therefore, the performance improvements observed under these strict evaluation settings cannot be simply attributed to output-format normalization.

We also audited the historical evaluation records and corrected inconsistencies in checkpoint selection and parser configurations to ensure consistent evaluation settings in the revised manuscript.

Response to Weakness 5

We provide the complete LoRA configuration and optimization settings used consistently across all training experiments.

LoRA and Optimization Configuration

ParameterConfiguration
Fine-tuning methodLoRA-based SFT
LoRA rank (rr)16
LoRA alpha32
LoRA dropout0
Target modulesq_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Adapted componentThinker LLM
Adapted layers (Omni-3B)All 36 layers
Adapted layers (Omni-7B)All 28 layers
OptimizerAdamW
Learning rate1×10−41\times10^{-4}
LR schedulerCosine
Warmup steps134
Gradient clipping1
PrecisionBF16
Training updates1,339

Trainable Parameters

ModelTrainable ParametersPercentage of Total Parameters
Qwen2.5-Omni-3B29.9M0.63%
Qwen2.5-Omni-7B40.4M0.45%

All training experiments use the same LoRA configuration and optimization settings.

Response to Weakness 6

We have revised Figure 7 by removing the ±0.5 percentage-point visual band to avoid potential confusion with confidence intervals. The updated figure is included in the revised manuscript.

Response to Weakness 7

We clarify the sources and semantics of emotion labels across different experiments:

  • Appendix F and Appendix H: CREMA-D and RAVDESS both use the intended emotions encoded in the original filenames. These labels are extracted during preprocessing and stored in the reference field of the evaluation manifest. Thus, both appendices use the same label set.
  • Representation Analysis: We use CREMA-D's official VoiceVote annotations, which represent emotions perceived by human listeners (perceived emotion), rather than emotions intended by the actors.
  • Other datasets: SAVEE uses the acted-emotion labels provided in its released metadata, while VocalSound uses its official non-linguistic vocalization categories. We will clarify the distinction between intended emotion and perceived emotion and specify the corresponding label sources and mappings for each experiment in the revised manuscript.

Response to Weakness 8

We have rechecked the experimental records and will correct the corresponding values throughout the revised manuscript.

  • Real–Silence Agreement: We correct the range reported in the Introduction from 60.1%–87.1% to 77.12%–90.90%.
  • Table 2: We correct the average-score description. 72.10 is the MMAU score, while the average across the three General Audio benchmarks is 69.00.
  • Tables 1 and 8: The two tables use different MMAU evaluation versions: MMAU test-mini in Table 1 and MMAU-v05.15.25 in Table 8. We will explicitly clarify the dataset versions and evaluation settings in the corresponding captions and text. We will ensure numerical consistency across the main text, tables, figures, and appendix in the revised manuscript.

Response to Weakness 9

We further examined the original layer-wise measurements and interpret Figure 8 in light of prior studies on audio information processing and decision formation in Transformers.

LayerBaseMixedMixed / Base
90.0021950.0021640.99×
270.0005590.03518563.0×
350.0011790.01807115.3×
  • Prior work [1] shows that audio information primarily shapes candidate representations in early-to-middle layers, while fine-tuning strengthens the contribution of candidate information to final answer selection in middle-to-late layers. Related research [2] further highlights the role of middle-to-late Transformer layers in task decision formation.
  • Consistent with these findings, we observe nearly unchanged representational differences at Layer 9 but substantially increased differences at Layer 27. This suggests that fine-tuning shifts more audio-conditioned representational changes toward middle-to-late layers, potentially allowing acoustic information to play a greater role in subsequent translation decisions.

References

[1] Tracing Audio Grounding and Answer Selection in Audio LLMs.

[2] Geometry of Decision Making in Language Models.

Overall Response and Additional Clarifications

Our central contribution is to identify and investigate the recognition–utilization gap in Omni models: recognizing vocal information does not necessarily mean that the model effectively uses it in subsequent decisions. Rather than introducing a new architecture or optimization objective, PCT demonstrates that this gap can be substantially mitigated through carefully constructed supervision and standard supervised fine-tuning.

Our additional experiments further strengthen this finding:

  • Supervision Complementarity: Matched-budget comparisons show that counterfactual supervision improves audio-conditioned translation decisions, while combining it with translation-quality supervision further improves both DF and translation-quality agreement.
  • Recognition–Utilization Analysis: Sample-level cross-analysis demonstrates that PCT substantially reduces the failure to use correctly recognized vocal meaning in translation decisions, providing direct behavioral evidence for the proposed recognition–utilization gap.
  • Task Generalization: Additional evaluations demonstrate improvements beyond the original translation-selection task, including emotion recognition and paralinguistic understanding.
  • Training and Evaluation Transparency: We have clarified the training configurations, checkpoint selection, evaluation protocols, emotion-label provenance, and numerical inconsistencies. We have also revised the relevant figures and corrected the corresponding experimental descriptions.

We also clarify that our current task focuses on audio-conditioned translation selection rather than open-ended translation generation. Extending the findings to free-form speech translation is an important direction for future work.

Overall, we believe the significance of this work lies not merely in the proposed fine-tuning procedure, but in identifying an underexplored limitation of Omni models and demonstrating a simple and effective way to address it. We hope this finding encourages further research on how multimodal models actually utilize the information they receive, rather than focusing exclusively on increasing model scale and training data.

Reviewer_mJ6eOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether multimodal AI models that process speech, text, and images actually understand how something is spoken, or whether they rely primarily on the words themselves. It introduces a controlled evaluation in which the same sentence is spoken with different vocal expressions, changing its intended meaning. The models must choose the translation that best reflects what the speaker conveys.

The study finds that even highly capable models largely ignore vocal expression, often making the same choice regardless of how the sentence is spoken. To address this weakness, the authors introduce Paralinguistic Counterfactual Training (PCT), which teaches models to recognize how changes in tone, emotion, and emphasis can alter meaning without changing the words.

The training substantially improves sensitivity to vocal expression while also improving translation quality. These gains extend to other speech-understanding tasks, with broader multimodal capabilities largely preserved. The central finding is that understanding the words of speech is not the same as understanding the speaker. Multimodal models can learn to use the expressive information in speech, but this ability requires deliberate training rather than emerging reliably from general language competence.

Critical strengths

The paper’s main contribution is a controlled method for measuring and improving whether multimodal models actually use paralinguistic information in speech, rather than relying on its transcript. It introduces a paired evaluation in which identical spoken words carry different meanings through changes in vocal expression, making it possible to distinguish genuine prosodic understanding from text-based guessing. Using this evaluation, the authors demonstrate that existing models largely neglect these cues despite performing well on conventional translation benchmarks.

To address this, the paper proposes Paralinguistic Counterfactual Training (PCT), a lightweight supervised fine-tuning approach that explicitly teaches models to associate changes in vocal expression with changes in meaning. The method substantially improves prosody-sensitive decisions, generalizes to other speech tasks, and largely preserves existing multimodal capabilities.

The central contribution is therefore both diagnostic and methodological: a rigorous way to expose models' reliance on transcripts, and a targeted training approach that makes vocal expression consequential to their decisions.

Strengths:

  1. The controlled diagnosis plus the Quality-only control is the real contribution: by holding transcript and candidates fixed and varying only audio, and by showing Quality-only raises human agreement while lowering DF, the authors demonstrate that aggregate quality/benchmark scores do not certify use of vocal information. Consequential because it undercuts a common evaluation assumption for omni models and gives a reusable, text-controlled probe.

  2. Breadth of validation makes the "not just probe-overfitting" case credible: gains replicate across 2 base + 4 post-trained omni models + a Gemma backbone, and PCT transfers to OOD emotion/audio benchmarks and largely preserves text/image/video/audiovisual performance and persists under noise — important because a cheap LoRA fix that degraded other modalities would not be adoptable.

Critical weaknesses
  1. The flagship number is partly in-distribution and only partial. PCT is trained on ContraProST counterfactual pairs and DF is evaluated on the ContraProST test split with the same format, so a large jump from supervised fine-tuning is partly expected; the genuinely OOD transfer is much smaller (general-audio Avg. gains ~1–3 pts; VocalSound ~0). DF plateaus near ~50%, meaning half of paired events still fail even in-distribution, the intervention mitigates but does not establish robust prosody grounding. The paper would be stronger if it sharply separated "learned the task format" from "learned to use prosody" (e.g., a held-out probe family with different prompting/metric).

  2. Scope of the central claim. "Omni models underuse how we speak" is supported only through one task family (MT-preference selection from ContraProST), en-source, 3 target languages, and a small test set (283 examples / 566 pairs). Whether the diagnosis and the fix generalize beyond this framing is asserted more than demonstrated; the layer-wise "mechanism" evidence is suggestive but correlational.

Questions for authors

Clarifications (could change assessment):

  1. What is DF on a prosody probe that differs in prompt format/metric from the training construction (to isolate format-learning from prosody use)? Does the ~50% ceiling correspond to systematically getting one audio condition right and its contrast wrong?

  2. Are the commercial models (Gemini-3.7-Flash, Qwen3.8-Omni-Flash, Qwen3.5-Omni-Plus) evaluated via API with audio truly ingested (not transcribed upstream)? How is TIE handled, and how sensitive are their DF numbers to prompt phrasing?

  3. Confirm no source-sentence or audio overlap between D_C/D_Q training and the DF test beyond the stated source-group stratification.

4 comments

Thank you for your insightful comment.

Response to Weakness 1

We further distinguish task-format learning from acoustic information utilization through prompt variations, a different semantic evaluation task, and an audio-removal control.

1. Prompt Robustness

PromptSettingOmni-3B PCTOmni-7B PCT
P0Original instruction + JSON47.3550.18
P1Reworded instruction + JSON47.0050.71
P2Original instruction + A/B/TIE44.8848.76
  • Rewording the instruction (P1) produces negligible changes in DF for both models.
  • Changing the output format (P2) reduces 3B DF by 2.5 pp (95% CI: [−4.8, −0.2]), while the change is not significant for 7B.
  • PCT retains a gain of over 40 pp over Base under all three prompts.

2. Four-view Case

Figure 1 illustrates a representative example in which two audio conditions share the same transcript but express different meanings, requiring different translation choices. The four-view evaluation further tests whether the model preserves the correct translation identity when candidate order is reversed.

Audio ConditionCandidate OrderGoldPCT Prediction
Audio 1A/BAA
Audio 1B/ABB
Audio 2A/BBB
Audio 2B/AAA

3. Meaning Probe

We further evaluate whether models can infer intended meaning directly from speech, without being given candidate translations.

ModelBase DFPCT DF
Omni-3B1.338.6
Omni-7B4.239.9

PCT substantially improves the interpretation of stress-induced meaning beyond the original candidate-selection format.

4. PCT-NoAudio Control

Training ConfigurationOmni-3B DF
Base1.59
PCT-NoAudio1.06
PCT47.35

PCT-NoAudio underperforms PCT by 46.29 pp (95% CI: [38.89, 53.55]), showing that text, candidate, and label supervision alone cannot reproduce the gains obtained from real acoustic supervision.

Together, these results show that PCT improves the use of acoustic information in translation decisions beyond learning a particular answer format.

Response to Weakness 2

We extended the evaluation to additional tasks and speech conditions to further examine the generalizability of PCT.

1. Cross-task and Real-speech Evaluation

EvaluationTaskOmni-3B GainOmni-7B Gain
EchoMindHuman Reasoning+2.52 pp+2.62 pp
EchoMindTTS Reasoning+2.91 pp+4.57 pp
StressTestStressed-word detection+31.2 pp+38.8 pp
StressTestStress interpretation+22.9 pp+9.2 pp
  • PCT improves both reasoning tasks on EchoMind across 3B and 7B models, demonstrating transfer beyond translation selection to independent paralinguistic reasoning tasks.
  • Results on real recorded speech from StressTest further show improvements in vocal-cue recognition and stress-induced meaning interpretation.
  • General Audio Understanding results in Table 1 provide additional evidence of PCT's performance across audio tasks.

Together, these results extend the evaluation of PCT to independent paralinguistic reasoning and real-speech understanding tasks.

2. Layer-wise Analysis

We follow the analytical perspective of [1], which uses layer-wise attention knockout to investigate two information pathways: Audio-to-Options and Options-to-Answer. Their findings show that audio information primarily shapes candidate representations in early-to-middle layers, while fine-tuning strengthens the contribution of candidate information to final answer prediction in middle-to-late layers.

Following this perspective, we re-examined our layer-wise representations and revised Figure 8 to use a shared absolute scale.

LayerBasePCTPCT / Base
90.0021950.0021640.99×
270.0005590.03518563.0×

Our results similarly show nearly unchanged representational differences in early layers but substantially increased audio-conditioned differences in middle-to-late layers. This observation is consistent with the findings in [1] regarding the role of middle-to-late layers in utilizing candidate information for decision-making.

We therefore present the layer-wise results as complementary evidence of changes in internal information processing and have revised the corresponding mechanistic interpretation.

[1] Tracing Audio Grounding and Answer Selection in Audio LLMs (2026).

Response to Question 1

The prompt robustness results, Meaning Probe, and PCT-NoAudio control are reported in Response to Weakness 1.

Regarding the approximately 50% DF, it is important to distinguish the strict pairwise criterion from its random baseline. Each of the two contrasting audio conditions has a 50% probability of receiving the correct translation under random binary selection. Since DF requires both conditions to be correct, its expected random baseline is:

P(DFrandom)=0.5×0.5=25%.P(\mathrm{DF}_{\mathrm{random}})=0.5\times0.5=25\%.

Thus, PCT's approximately 50% DF is twice the expected random baseline, demonstrating that the model uses acoustic contrasts to adjust its translation decisions, while leaving room for further improvement.

We further decompose the paired outcomes into four categories:

Audio Condition 1Audio Condition 2Outcome
CorrectCorrectBoth correct
CorrectWrongOnly condition 1 correct
WrongCorrectOnly condition 2 correct
WrongWrongNeither correct

This analysis distinguishes one-sided failures from cases where both contrasting conditions are incorrectly handled, providing a more detailed account of the remaining DF errors.

Response to Question 2

  • Audio Input: Gemini-3.7-Flash, Qwen3.8-Omni-Flash, and Qwen3.5-Omni-Plus were all evaluated through their respective APIs with raw audio inputs, rather than upstream ASR transcripts. The original API request records confirm that audio data was included.
  • TIE Handling: TIE responses were excluded from the valid evaluation denominator when computing DF, following the original evaluation protocol.
  • Prompt Consistency: All three commercial models used the same task instructions and candidate-selection format as Figure 2 and the subsequent evaluations in the paper. Due to API usage constraints, we conducted the P0/P1/P2 prompt-robustness experiments only on the local Omni-3B and Omni-7B models. The results are reported in Response to Weakness 1.

Response to Question 3

We audited the training datasets against the final DF evaluation set and confirmed that there is no overlap at either the source-sentence or audio level.

Training DataSource-sentence OverlapAudio Overlap
DCD_C (ContraProST)00
DQD_Q (IWSLT)00

Both training datasets are therefore disjoint from the final DF evaluation set in terms of source sentences and audio content.

Dear Reviewer mJ6e,

We sincerely appreciate your time and effort in reviewing our manuscript and offering valuable suggestions.

As the author-reviewer discussion phase is drawing to a close, we would like to confirm whether our responses have effectively addressed your concerns.

We provided detailed responses to your concerns a few days ago, and we hope they have adequately addressed your issues. lf you require further clarification or have any additional concerns, please do not hesitate to contact us. We are more than willing to continue our communication with you.

With a few hours remaining in the discussion period, we would be grateful for your thoughts on our response and revised manuscript if you have a chance to share them. We would be happy to discuss any questions you may still have.

Thank you again for your time.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Summary

We greatly appreciate all reviewers for their constructive feedback and insightful suggestions, which have helped us improve the clarity, presentation, and experimental validation of our work. We have carefully addressed the raised concerns, conducted additional experiments, and revised the manuscript to better clarify our research motivation and contributions.

Our work investigates an important question in omni models: whether models actually utilize the multimodal information they receive to guide their decisions. While recent advances have substantially improved multimodal architectures and capabilities, our study reveals that the ability to receive and encode audio information does not necessarily translate into its effective use in subsequent decision-making.

Starting from controlled speech translation, we identify a recognition–utilization gap across evaluated open-source and commercial omni models. To address this issue, we introduce Paralinguistic Counterfactual Training (PCT), a lightweight supervision approach that combines balanced same-transcript audio pairs with contrasting target choices and human translation-quality preferences. PCT uses standard supervised fine-tuning without modifying the model architecture or introducing additional complex optimization objectives.

Our experiments demonstrate that PCT substantially improves audio-conditioned translation decisions, increasing paired correctness from 2.65% to 49.29% on Omni-3B and from 8.17% to 50.71% on Omni-7B. Matched-budget ablations further establish the benefit of jointly supervising contrasting audio conditions over single-condition training. Additional evaluations demonstrate transfer to general audio understanding, emotion recognition, and paralinguistic tasks, including independent real-speech evaluations. We also examine training variability across independent runs and the retention of capabilities in other modalities.

Beyond the proposed method, we believe this work highlights an important and underexplored direction for omni model research: moving beyond the ability to process multimodal inputs toward understanding and improving how models actually utilize multimodal information in their decisions. The proposed approach has also been applied in practical business scenarios, demonstrating its applicability beyond benchmark evaluation.

We have incorporated the additional analyses and corresponding manuscript revisions to make these contributions clearer. We sincerely thank the reviewers, AC, SAC, and PC for their time and consideration.

Sincerely,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.