- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the gap between linguistic and paralinguistic understanding in Speech LLMs. The authors construct PARAtaxis, a dataset of acoustically verified speech pairs with identical transcripts and speaker identities but different prosodic characteristics, including speaking rate, energy, pitch, and intonation. Through controlled evaluations of nine pretrained Speech LLMs and layer-wise representation analysis, they identify a substantial gap between lexical and paralinguistic discrimination.
To address this issue, the authors propose ParaNCE, a bidirectional speech–caption contrastive learning objective that explicitly incorporates paralinguistic hard negatives and is jointly optimized with next-token prediction. Experiments on Qwen2.5-Omni-7B and Audio Flamingo 3 demonstrate improvements in controlled paralinguistic discrimination. Further evaluations on MMSU and five emotion-recognition datasets show gains in downstream speech understanding and transfer to emotion recognition without explicit emotion-category supervision.
Critical strengths
1. Well-motivated problem formulation and controlled evaluation. The paper identifies an important limitation of Speech LLMs in distinguishing paralinguistic differences. By constructing speech pairs with identical transcripts and speaker identities but different acoustic characteristics, the authors provide a controlled setting for investigating this limitation. The analysis across multiple models and representation layers further strengthens the motivation for the proposed approach.
2. Improvements in downstream paralinguistic understanding. The proposed method demonstrates improvements on MMSU, particularly in perception tasks, compared with both the pretrained model and conventional fine-tuning baselines. The evaluation on multiple emotion-recognition datasets also provides evidence that learning foundational acoustic cues can benefit tasks beyond those directly supervised during training.
Critical weaknesses
1. Limited and inconsistent advantages over existing adaptation methods. Although ParaNCE improves MMSU performance, its additional gains over Dense SFT and InfoNCE are relatively modest, particularly on reasoning tasks. Moreover, the improvements on the second model, Audio Flamingo 3, are considerably smaller, raising concerns about the consistency of the method across different model architectures. InfoNCE also consistently outperforms ParaNCE across all five emotion-recognition datasets, despite cross-task transfer being emphasized as a major contribution. Furthermore, the evaluation lacks comparisons with existing preference-optimization or reinforcement-learning-based approaches to paralinguistic understanding. These limitations make it difficult to assess the general effectiveness and necessity of the proposed contrastive objective. 2. Oversimplified paralinguistic attributes and insufficient evaluation of general capabilities. The proposed approach focuses on only four basic acoustic attributes: speaking rate, energy, pitch, and intonation. While these attributes are measurable and useful, they represent a relatively narrow subset of paralinguistic information and do not fully capture more complex phenomena such as sarcasm, speaker intent, or context-dependent vocal expressions. Furthermore, since the method jointly fine-tunes the audio encoder and language decoder, it remains unclear whether improvements in these specific tasks come at the expense of general speech understanding or other capabilities. Evaluations beyond paralinguistic benchmarks would help establish the broader applicability of the method.
Questions for authors
1. Effectiveness across different models and downstream tasks. The improvements on Audio Flamingo 3 are substantially smaller than those on Qwen2.5-Omni-7B. Additionally, standard InfoNCE outperforms ParaNCE on all five emotion-recognition datasets. Could the authors explain these differences and provide further evidence that the proposed contrastive objective offers consistent advantages across model architectures and downstream tasks?
2. Comparison with existing optimization approaches. Could the authors provide comparisons with existing preference-optimization or reinforcement-learning-based approaches to paralinguistic understanding, such as VoxParadox, under comparable experimental settings? Such comparisons would help clarify the effectiveness of representation-level contrastive learning relative to alternative optimization strategies.
3. Generalization beyond basic acoustic attributes. The training data focuses on four basic acoustic attributes: speaking rate, energy, pitch, and intonation. Is there evidence that the proposed method improves the understanding of more complex paralinguistic phenomena, such as sarcasm, speaker intent, or context-dependent vocal expressions? Such evidence would help establish whether the method learns more general paralinguistic representations rather than primarily improving discrimination of the selected acoustic attributes.
4. Impact on general speech and language capabilities. Since the proposed method jointly fine-tunes the audio encoder and language decoder, have the authors evaluated whether it affects general capabilities such as speech recognition, semantic understanding, and instruction following? Results on general-purpose benchmarks would help determine whether the paralinguistic improvements are achieved without compromising the model's existing capabilities.
13 comments
Replies to Reviewer 7991 – Review 1, Part II
Model Differences and the Original AF3 Training Setup
Beyond the task-level explanation for reasoning gains, the smaller improvements on AF3 also warrant consideration of backbone differences. Qwen2.5-Omni [1] and AF3 [2] use different audio encoders and training regimes, which may affect their responses to the same adaptation settings. For our initial validation on AF3, we directly adopted Qwen2.5-Omni’s training recipe and hyperparameters without AF3-specific tuning, which we will clarify in the appendix. To investigate whether training configurations contribute to the differences in improvements across models, we conducted supplementary experiments varying the candidate-pool size and learning rate.
Comparing Gains after Tuning AF3
Specifically, we tested candidate-pool sizes of 4, 8, and 16 at learning rates of 3 × 10⁻⁶ and 1 × 10⁻⁶, alongside Dense SFT baselines at each learning rate under the same training budget and evaluation protocol. The original paper setting uses 4 candidates at 3 × 10⁻⁶. The table reports MMSU accuracy (%).
| AF3 method | Learning rate | Pool size | Overall | Perception | Reasoning |
|---|---|---|---|---|---|
| Base | — | — | 59.46 | 43.95 | 75.99 |
| Dense SFT | 3×10⁻⁶ | — | 60.32 | 45.62 | 75.99 |
| ParaNCE | 4 | 61.16 | 47.09 | 76.16 | |
| 8 | 62.02 | 48.57 | 76.36 | ||
| 16 | 61.30 | 47.71 | 75.79 | ||
| Dense SFT | 1×10⁻⁶ | — | 59.94 | 44.11 | 76.82 |
| ParaNCE | 4 | 60.06 | 44.30 | 76.86 | |
| 8 | 60.28 | 44.73 | 76.86 | ||
| 16 | 60.38 | 45.04 | 76.74 |
The strongest Overall and Perception results in this comparison are obtained with eight candidates at a learning rate of 3 × 10⁻⁶. Compared with the original setting of four candidates at the same learning rate, Overall accuracy increases by 0.86 percentage points (p = 0.042) and Perception by 1.47 points (p = 0.030), with a small improvement in Reasoning.
Using this AF3 configuration, we further separate the gains from Dense SFT over Base and the additional gains from ParaNCE across both backbones. All values are in percentage points (pp).
| Metric | Within-backbone comparison | Qwen2.5-Omni gain | AF3 gain | Gap (Qwen − AF3) |
|---|---|---|---|---|
| Overall | Dense SFT − Base | +3.60 | +0.86 | 2.74 |
| ParaNCE − Dense SFT | +1.72 | +1.70 | 0.02 | |
| Perception | Dense SFT − Base | +4.61 | +1.67 | 2.94 |
| ParaNCE − Dense SFT | +3.10 | +2.95 | 0.15 | |
| Reasoning | Dense SFT − Base | +2.52 | 0.00 | 2.52 |
| ParaNCE − Dense SFT | +0.25 | +0.37 | −0.12 |
The cross-backbone difference in total adaptation gains arises primarily from Dense SFT rather than ParaNCE. On Overall, Dense SFT improves Qwen2.5-Omni by 3.60 pp versus 0.86 pp on AF3, creating a 2.74 pp gap that accounts for nearly all of the 2.76 pp difference in total gains. A similar pattern appears in Perception, where Dense SFT contributes a 2.94 pp gap (4.61 vs. 1.67 pp), compared with a 3.09 pp difference in total gains. Reasoning follows the same pattern, with Dense SFT gains of +2.52 pp on Qwen2.5-Omni and 0.00 pp on AF3.
In contrast, ParaNCE provides nearly identical additional gains over Dense SFT on Overall (+1.72 vs. +1.70 pp) and Perception (+3.10 vs. +2.95 pp), with statistically significant improvements on both backbones (all p < 0.01). Its smaller Reasoning gains are also comparable (+0.25 vs. +0.37 pp). These results indicate that the smaller total gains on AF3 largely reflect differences in Dense SFT improvements, while supporting the consistency of ParaNCE's additional benefits across backbones.
We hope these clarifications and supplementary experiments address the reviewer’s concerns about reasoning gains and cross-model consistency. Together, the results support the effectiveness of our focused paralinguistic supervision and demonstrate comparable additional benefits from ParaNCE across the two backbones.
References
[1] Qwen2.5-Omni Technical Report, arXiv, 2025.
[2] Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models, NeurIPS, 2025.
Replies to Reviewer 7991 – Review 2
Summarized Review 2: Performance Gap on Emotion Tasks
We thank the reviewer for raising W1 and Q1, which we summarize as asking why InfoNCE consistently outperforms ParaNCE across all five emotion-recognition benchmarks, particularly on EMO-DB, and whether ParaNCE’s contrastive objective nevertheless provides additional benefits for broader paralinguistic understanding. To address these concerns, we conducted a sampling ablation and extended our evaluation to paralinguistic tasks beyond emotion recognition.
InfoNCE benefits from our paired sampling
Our paper's InfoNCE baseline uses ParaNCE's hard-pair sampling, although InfoNCE does not require it [1]. Under this strategy, a contrastive pool of size contains pairs with identical transcripts and speaker identities but different paralinguistic attributes. To assess the sampling benefit, we trained InfoNCE w/o Pairing with random negatives. The table reports accuracy (%), with parentheses indicating percentage-point changes from InfoNCE + Pairing.
| Method | CREMA-D | IEMOCAP | ESD-EN | EMO-DB | RAVDESS |
|---|---|---|---|---|---|
| Base | 27.72 | 63.52 | 29.93 | 47.66 | 21.25 |
| Dense SFT | 31.98 | 67.08 | 37.38 | 54.02 | 33.26 |
| InfoNCE + Pairing | 45.54 | 69.56 | 51.81 | 69.53 | 44.51 |
| InfoNCE w/o Pairing | 29.20 (−16.34) | 66.26 (−3.30) | 42.58 (−9.23) | 53.64 (−15.89) | 32.36 (−12.15) |
| ParaNCE | 40.09 | 66.76 | 39.87 | 50.65 | 34.17 |
The ablation shows that our hard-pair sampling provides substantial and unexpected benefits for emotion recognition, as removing pairing reduces InfoNCE accuracy by 3.30–16.34 points across all five datasets. More broadly, all four adaptations improve acoustic perception and achieve significant mean emotion-recognition gains over Base (+6.73–18.17 points; two-sided paired t-tests, , all unadjusted ), including an 8.29-point gain for ParaNCE (). These findings support the transfer from learning basic acoustic attributes to higher-level emotion understanding. The differences across training objectives further suggest that this transfer may depend on how each objective shapes acoustic representations, which we are continuing to investigate. See more details in Part 2 of our response to reviewer WPxz.
Broader Benefits of Task Transferring
To further establish ParaNCE's advantages in paralinguistic tasks transferring, we extended our evaluation on MMSU [2] and MSPB [3]. The table reports overall accuracy (%).
| Dataset | Task | N | Base | InfoNCE + Pairing | InfoNCE w/o Pairing | ParaNCE |
|---|---|---|---|---|---|---|
| MMSU | Sarcasm detection | 116 | 71.55 | 73.28 | 68.97 | 71.55 |
| Intonation-based reasoning | 78 | 70.51 | 70.51 | 74.36 | 70.51 | |
| Prolonged-sound reasoning | 84 | 95.24 | 96.43 | 96.43 | 97.62 | |
| Pause-based reasoning | 90 | 52.22 | 58.89 | 56.67 | 62.22 | |
| Stress-based reasoning | 110 | 43.64 | 43.64 | 42.73 | 45.45 | |
| Emotion recognition | 117 | 35.90 | 53.85 | 42.74 | 54.70 | |
| Emotional-context reasoning | 110 | 20.91 | 27.27 | 23.64 | 30.00 | |
| Overall | 705 | 53.62 | 58.87 | 55.74 | 60.00 | |
| MSPB | Statement/question intonation | 40 | 57.50 | 75.00 | 62.50 | 77.50 |
| Ambiguity resolution | 40 | 60.00 | 57.50 | 60.00 | 60.00 | |
| Prosodic focus | 24 | 29.17 | 29.17 | 29.17 | 29.17 | |
| Focus operators | 14 | 50.00 | 71.43 | 57.14 | 71.43 | |
| Scalar meaning | 10 | 50.00 | 50.00 | 60.00 | 50.00 | |
| Irony comprehension | 10 | 80.00 | 70.00 | 70.00 | 70.00 | |
| Emotion recognition without context | 32 | 46.88 | 50.00 | 46.88 | 59.38 | |
| Emotion recognition with context | 8 | 37.50 | 37.50 | 37.50 | 50.00 | |
| Overall | 178 | 51.69 | 56.74 | 53.37 | 60.11 |
Results show that ParaNCE achieves the highest overall accuracy on both benchmarks and leads or ties on 11 of the 15 tasks, indicating that our proposed ParaNCE benefits transfer beyond the supervised acoustic tasks to broader paralinguistic understanding.
Together, these findings highlight the substantial contribution of our proposed method. The unexpected benefit on InfoNCE also motivates further investigation into how sampling strategies and training objectives jointly shape paralinguistic learning and transfer. Meanwhile, the extended evaluations provide further evidence that ParaNCE's benefits transfer to broader paralinguistic tasks. We hope these findings adequately address concerns raised by the reviewer.
References
[1] Representation Learning with Contrastive Predictive Coding, arXiv preprint arXiv:1807.03748, 2018.
[2] MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark, ICLR, 2026.
[3] Can AI Understand Mandarin Speech Prosody? A Framework and Benchmark Showcase, Interspeech, 2025.
Replies to Reviewer 7991 – Review 1, Part I
Summarized Review 1: Modest Reasoning Gains and Differences Across Model Architectures
We thank the reviewer for raising W1 and Q1 regarding ParaNCE’s relatively modest gains over Dense SFT and InfoNCE on reasoning tasks, and its smaller improvements on Audio Flamingo 3 (AF3) than on Qwen2.5-Omni-7B. We address these concerns by examining the scope of our training supervision, differences between the backbones, and additional experiments on AF3's training configuration.
Reasoning Gains in Relation to the Training Tasks
We designed PARAtaxis to strengthen foundational paralinguistic perception through verifiable acoustic supervision. This deliberate focus on acoustic understanding provides the basis for interpreting its gains on broader reasoning tasks. Its seven task formats primarily target the perception and description of speaking rate, energy, pitch, and intonation, as illustrated by the following simplified examples.
While these tasks directly supervise paralinguistic perception, MMSU reasoning tasks additionally require interpreting acoustic evidence in relation to semantic and contextual information. The examples below illustrate this distinction.
This distinction helps explain the relatively modest reasoning gains. While improved paralinguistic perception provides acoustic evidence for contextual interpretation, our training does not directly supervise the semantic and contextual inferences in MMSU reasoning tasks. The substantial gains in perception, alongside preserved or modestly improved reasoning performance, therefore support the effectiveness of our training design.