- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether omni models use paralinguistic cues, such as prosody and emotion, when making decisions. The authors introduce a controlled translation-selection task in which recordings with identical transcripts but different vocal expressions support different translations. They find that several omni models rarely make the correct audio-dependent choice. To address this, they propose Paralinguistic Counterfactual Training, which combines counterfactual audio supervision with human translation-quality preferences. PCT substantially improves paired correctness and shows transfer to general audio understanding and paralinguistic tasks.
Critical strengths
-
A well-motivated diagnostic. The controlled evaluation distinguishes general translation-quality judgment from the ability to use vocal information when it changes the correct answer. The silence control provides additional evidence of this gap.
-
A simple and effective intervention. PCT uses standard supervised fine-tuning with LoRA. The quality-only and counterfactual-only ablations support combining both supervision sources.
-
Broad evaluation. Experiments cover multiple model families, audio understanding benchmarks, paralinguistic tasks, and cross-modal capability retention.
Critical weaknesses
-
The contribution of mixed supervision is not fully isolated. The paper does not specify whether the single-source controls match PCT in training steps and example exposure, and Counter-only results on the main DF metric are missing. Numerical training settings, including the learning rate and LoRA configuration, are also needed to assess and reproduce these comparisons.
-
DF provides evidence of audio-dependent decisions, but leaves candidate-order robustness unclear. It averages paired correctness separately over the two candidate orders rather than requiring all four views to be correct. Reporting order consistency alongside DF would clarify whether the improved choices reliably survive candidate reversal.
Questions for authors
-
Could you report candidate-order consistency, all-four-view correctness, and TIE/invalid-output rates before and after PCT?
-
What DF does Counter-only achieve? Were the single-source controls matched to PCT in training steps and example exposure? Please also provide the numerical training hyperparameters and checkpoint-selection criteria.
-
Can the transfer results be validated on an independent benchmark involving natural conversational speech or unseen paralinguistic phenomena? For cross-modal retention, could you also provide uncertainty estimates or formal non-inferiority tests?
4 comments
Response to Question 3
We provide additional evaluations on independent real speech and cross-modal retention to examine PCT's transferability and its impact on other modalities.
1. Independent Real-speech Evaluation
We evaluated PCT on the independent StressTest benchmark using real recorded speech, assessing stressed-word detection and stress interpretation.
| Task | Model | Base | PCT | Improvement | 95% CI |
|---|---|---|---|---|---|
| Stressed-word Detection | Omni-3B | 21.80% | 52.90% | +31.2 pp | [24.8, 37.7] |
| Stress Interpretation | Omni-3B | 40.40% | 63.30% | +22.9 pp | [16.4, 29.6] |
| Stressed-word Detection | Omni-7B | 15.30% | 54.10% | +38.8 pp | [31.1, 46.7] |
| Stress Interpretation | Omni-7B | 54.60% | 63.80% | +9.2 pp | [3.7, 14.8] |
All four improvements have 95% bootstrap confidence intervals entirely above zero, demonstrating that PCT's improvements transfer to an independent real-speech evaluation.
2. Cross-modal Retention
We further evaluated whether audio-focused post-training affects the model's existing capabilities in other modalities, including text, video, vision, and audio-visual understanding.
We performed paired evaluations on identical test examples and calculated 95% confidence intervals using 10,000 paired bootstrap samples.
| Model | Benchmark | Base | PCT | Δ (pp) | 95% CI |
|---|---|---|---|---|---|
| 3B | MMLU-Redux | 55.77 | 56.69 | 0.92 | [−0.22, +2.09] |
| 3B | MLVU | 63.80 | 62.78 | −1.01 | [−2.02, −0.05] |
| 3B | MMStar | 54.60 | 54.87 | 0.27 | [−1.07, +1.67] |
| 3B | DailyOmni | 58.19 | 57.19 | −1.00 | [−2.46, +0.44] |
| 7B | MMLU-Redux | 66.57 | 66.39 | −0.18 | [−1.03, +0.66] |
| 7B | MLVU | 65.96 | 66.56 | 0.60 | [−0.22, +1.43] |
| 7B | MMStar | 59.40 | 60.60 | 1.20 | [−0.13, +2.54] |
| 7B | DailyOmni | 61.54 | 61.71 | 0.17 | [−1.18, +1.48] |
- Omni-7B: None of the four cross-modal performance differences is statistically significant.
- Omni-3B: Three of the four differences are not statistically significant, while MLVU shows a small decrease of 1.01 pp.
Overall, these results suggest that PCT's audio-focused post-training largely preserves the model's existing capabilities across other modalities, without introducing consistent cross-modal degradation.
Dear Reviewer RvWR,
We sincerely appreciate your time and effort in reviewing our manuscript and offering valuable suggestions.
As the author-reviewer discussion phase is drawing to a close, we would like to confirm whether our responses have effectively addressed your concerns.
We provided detailed responses to your concerns a few days ago, and we hope they have adequately addressed your issues. lf you require further clarification or have any additional concerns, please do not hesitate to contact us. We are more than willing to continue our communication with you.
Thank you for the detailed responses and additional analyses. They address my main concerns about the training setup, candidate-order robustness, and transfer to independent real-speech data. Overall, I maintain my original rating 3: Weak acceptance.
Thank you for your insightful comment.
Response to Weakness 1
We provide additional details on the training budgets, performance comparisons, and numerical configurations of single-source and mixed supervision.
1. Training Budget
All supervision configurations use identical training budgets and optimization settings.
2. Counter-only vs. Mixed PCT
Compared with Counter-only, Mixed PCT improves DF by 9.93 pp and 9.44 pp on Omni-3B and Omni-7B, respectively. It also improves IWSLT HA by 5.10 pp and 7.38 pp, respectively.
These results demonstrate that combining counterfactual audio supervision with human translation-quality preferences improves both audio-conditioned translation decisions and general translation quality.
3. Numerical Training Configuration
All training experiments use the same LoRA and optimization configurations.
q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_projadapter_last)Response to Weakness 2 and Question 1
We provide additional analyses of candidate-order consistency and all-four-view accuracy to further evaluate the stability of PCT's decisions under candidate-order reversal.
1. Candidate-order Consistency (Omni-7B)
Order consistency is calculated based on translation identity across AB/BA candidate orders, rather than directly comparing output letters.
2. All-four-view Accuracy (Omni-3B)
These results demonstrate that PCT improves not only audio-conditioned translation decisions but also the consistency of these decisions under candidate-order reversal.
3. TIE and Invalid Outputs
For Omni-7B, Base produces 25 TIE outputs, whereas PCT produces neither TIE nor invalid outputs. Neither configuration produces invalid outputs.
Response to Question 2
Counter-only Performance: Counter-only achieves DF scores of 39.36% and 41.27% on Omni-3B and Omni-7B, respectively. PCT combines two equally weighted supervision sources: translation-quality supervision (DQ) from IWSLT and counterfactual audio supervision (DC) from ContraProST. DQ promotes translation-quality judgments, while DC teaches the model to select translations according to vocal conditions. Their combination achieves higher DF scores of 49.29% and 50.71%, respectively, while also improving IWSLT human agreement. See the performance comparison in our response to Weakness 1.
Training Budget: Quality-only, Counter-only, and Mixed PCT use the same total training exposure of 10,712 presentations, 1,339 optimizer updates, and 8 rows per update, with the same learning rate 1e−4 and cosine schedule. See the training-budget comparison in our response to Weakness 1.
Training Hyperparameters: All experiments use the same LoRA and optimization configurations, including rank 16, alpha 32, dropout 0, AdamW, and BF16. The complete configuration is provided in our response to Weakness 1.
Checkpoint Selection: All experiments use the terminal checkpoint (
adapter_last) after 1,339 optimizer updates, without dev-set-based checkpoint selection.