acceptodds
Under review as a conference paper at ICLR 2027

Leveraging Contrastive Cues for Paralinguistic Understanding in Speech LLMs

Abstract

How Speech LLMs distinguish paralinguistic information remains poorly understood, with limited resources for studying foundational cues independently of linguistic content. We first introduce ParaCorpus, a large-scale parallel speech–descriptor corpus built using text-to-speech synthesis and verified against acoustic criteria for four foundational dimensions: speaking rate, energy, pitch, and intonation. Its paired utterances with the same words but different delivery enable controlled behavioral tests and layerwise cosine-similarity analysis of speech–descriptor matching. Results reveal strong linguistic but weak paralinguistic discrimination. Persisting disparity despite supervised fine-tuning on fine-grained annotations indicates a systematic flaw over paralinguistic understanding. We therefore propose ParaDisc, a representation-level contrastive framework jointly optimizing hard-negative speech–descriptor matching and next-token prediction. ParaDisc strengthens representation-level paralinguistic matching and behavioral discrimination, improving MMSU perception and reasoning accuracy by 2.32 and 0.78 percentage over Qwen2.5-Omni-7B, respectively. Notably, it also improves emotion recognition without emotion supervision, emerging a 4.34% gain on IEMOCAP. These findings suggest that learning to distinguish foundational paralinguistic cues through speech–descriptor matching can support paralinguistic understanding, providing an empirical basis for disentangling how these cues contribute to higher-level affective.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Speech Large Language Models, Paralinguistic Understanding, Contrastive Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7991OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the gap between linguistic and paralinguistic understanding in Speech LLMs. The authors construct PARAtaxis, a dataset of acoustically verified speech pairs with identical transcripts and speaker identities but different prosodic characteristics, including speaking rate, energy, pitch, and intonation. Through controlled evaluations of nine pretrained Speech LLMs and layer-wise representation analysis, they identify a substantial gap between lexical and paralinguistic discrimination.

To address this issue, the authors propose ParaNCE, a bidirectional speech–caption contrastive learning objective that explicitly incorporates paralinguistic hard negatives and is jointly optimized with next-token prediction. Experiments on Qwen2.5-Omni-7B and Audio Flamingo 3 demonstrate improvements in controlled paralinguistic discrimination. Further evaluations on MMSU and five emotion-recognition datasets show gains in downstream speech understanding and transfer to emotion recognition without explicit emotion-category supervision.

Critical strengths

1. Well-motivated problem formulation and controlled evaluation. The paper identifies an important limitation of Speech LLMs in distinguishing paralinguistic differences. By constructing speech pairs with identical transcripts and speaker identities but different acoustic characteristics, the authors provide a controlled setting for investigating this limitation. The analysis across multiple models and representation layers further strengthens the motivation for the proposed approach.

2. Improvements in downstream paralinguistic understanding. The proposed method demonstrates improvements on MMSU, particularly in perception tasks, compared with both the pretrained model and conventional fine-tuning baselines. The evaluation on multiple emotion-recognition datasets also provides evidence that learning foundational acoustic cues can benefit tasks beyond those directly supervised during training.

Critical weaknesses

1. Limited and inconsistent advantages over existing adaptation methods. Although ParaNCE improves MMSU performance, its additional gains over Dense SFT and InfoNCE are relatively modest, particularly on reasoning tasks. Moreover, the improvements on the second model, Audio Flamingo 3, are considerably smaller, raising concerns about the consistency of the method across different model architectures. InfoNCE also consistently outperforms ParaNCE across all five emotion-recognition datasets, despite cross-task transfer being emphasized as a major contribution. Furthermore, the evaluation lacks comparisons with existing preference-optimization or reinforcement-learning-based approaches to paralinguistic understanding. These limitations make it difficult to assess the general effectiveness and necessity of the proposed contrastive objective. 2. Oversimplified paralinguistic attributes and insufficient evaluation of general capabilities. The proposed approach focuses on only four basic acoustic attributes: speaking rate, energy, pitch, and intonation. While these attributes are measurable and useful, they represent a relatively narrow subset of paralinguistic information and do not fully capture more complex phenomena such as sarcasm, speaker intent, or context-dependent vocal expressions. Furthermore, since the method jointly fine-tunes the audio encoder and language decoder, it remains unclear whether improvements in these specific tasks come at the expense of general speech understanding or other capabilities. Evaluations beyond paralinguistic benchmarks would help establish the broader applicability of the method.

Questions for authors

1. Effectiveness across different models and downstream tasks. The improvements on Audio Flamingo 3 are substantially smaller than those on Qwen2.5-Omni-7B. Additionally, standard InfoNCE outperforms ParaNCE on all five emotion-recognition datasets. Could the authors explain these differences and provide further evidence that the proposed contrastive objective offers consistent advantages across model architectures and downstream tasks?

2. Comparison with existing optimization approaches. Could the authors provide comparisons with existing preference-optimization or reinforcement-learning-based approaches to paralinguistic understanding, such as VoxParadox, under comparable experimental settings? Such comparisons would help clarify the effectiveness of representation-level contrastive learning relative to alternative optimization strategies.

3. Generalization beyond basic acoustic attributes. The training data focuses on four basic acoustic attributes: speaking rate, energy, pitch, and intonation. Is there evidence that the proposed method improves the understanding of more complex paralinguistic phenomena, such as sarcasm, speaker intent, or context-dependent vocal expressions? Such evidence would help establish whether the method learns more general paralinguistic representations rather than primarily improving discrimination of the selected acoustic attributes.

4. Impact on general speech and language capabilities. Since the proposed method jointly fine-tunes the audio encoder and language decoder, have the authors evaluated whether it affects general capabilities such as speech recognition, semantic understanding, and instruction following? Results on general-purpose benchmarks would help determine whether the paralinguistic improvements are achieved without compromising the model's existing capabilities.

13 comments
AuthorsOpenReview ↗

Replies to Reviewer 7991 – Review 1, Part I

Summarized Review 1: Modest Reasoning Gains and Differences Across Model Architectures

We thank the reviewer for raising W1 and Q1 regarding ParaNCE’s relatively modest gains over Dense SFT and InfoNCE on reasoning tasks, and its smaller improvements on Audio Flamingo 3 (AF3) than on Qwen2.5-Omni-7B. We address these concerns by examining the scope of our training supervision, differences between the backbones, and additional experiments on AF3's training configuration.

Reasoning Gains in Relation to the Training Tasks

We designed PARAtaxis to strengthen foundational paralinguistic perception through verifiable acoustic supervision. This deliberate focus on acoustic understanding provides the basis for interpreting its gains on broader reasoning tasks. Its seven task formats primarily target the perception and description of speaking rate, energy, pitch, and intonation, as illustrated by the following simplified examples.

Training taskExample questionTarget response
Pairwise comparisonWhich segment is louder?Segment 2.
Attribute explanationDescribe the intonation.Wavy contour.
Claim correctionIs the speaking rate slow?No, moderate.
Speech captioningDescribe the vocal delivery.Moderate rate, soft energy, high pitch, wavy intonation.
Content attributionWhat is said in the louder segment?"Montreal is a bilingual city."
Segment localizationWhich segment has lower pitch?Segment 2.
Pattern selectionSelect the intonation pattern.Rising–wavy–falling.

While these tasks directly supervise paralinguistic perception, MMSU reasoning tasks additionally require interpreting acoustic evidence in relation to semantic and contextual information. The examples below illustrate this distinction.

Reasoning taskExample questionRequirement
Intonation-basedWhich clip expresses reassurance?Infer communicative intent from vocal delivery.
Emotional contextWhat situation likely caused the speaker's emotion?Relate vocal emotion to situational context.
DialogueWhat is the speakers' relationship?Infer social relationships from dialogue.
CausalWhat causal relationship is implied?Infer causality from spoken meaning.

This distinction helps explain the relatively modest reasoning gains. While improved paralinguistic perception provides acoustic evidence for contextual interpretation, our training does not directly supervise the semantic and contextual inferences in MMSU reasoning tasks. The substantial gains in perception, alongside preserved or modestly improved reasoning performance, therefore support the effectiveness of our training design.

AuthorsOpenReview ↗

Replies to Reviewer 7991 – Review 1, Part II

Model Differences and the Original AF3 Training Setup

Beyond the task-level explanation for reasoning gains, the smaller improvements on AF3 also warrant consideration of backbone differences. Qwen2.5-Omni [1] and AF3 [2] use different audio encoders and training regimes, which may affect their responses to the same adaptation settings. For our initial validation on AF3, we directly adopted Qwen2.5-Omni’s training recipe and hyperparameters without AF3-specific tuning, which we will clarify in the appendix. To investigate whether training configurations contribute to the differences in improvements across models, we conducted supplementary experiments varying the candidate-pool size and learning rate.

Comparing Gains after Tuning AF3

Specifically, we tested candidate-pool sizes of 4, 8, and 16 at learning rates of 3 × 10⁻⁶ and 1 × 10⁻⁶, alongside Dense SFT baselines at each learning rate under the same training budget and evaluation protocol. The original paper setting uses 4 candidates at 3 × 10⁻⁶. The table reports MMSU accuracy (%).

AF3 methodLearning ratePool sizeOverallPerceptionReasoning
Base——59.4643.9575.99
Dense SFT3×10⁻⁶—60.3245.6275.99
ParaNCE461.1647.0976.16
862.0248.5776.36
1661.3047.7175.79
Dense SFT1×10⁻⁶—59.9444.1176.82
ParaNCE460.0644.3076.86
860.2844.7376.86
1660.3845.0476.74

The strongest Overall and Perception results in this comparison are obtained with eight candidates at a learning rate of 3 × 10⁻⁶. Compared with the original setting of four candidates at the same learning rate, Overall accuracy increases by 0.86 percentage points (p = 0.042) and Perception by 1.47 points (p = 0.030), with a small improvement in Reasoning.

Using this AF3 configuration, we further separate the gains from Dense SFT over Base and the additional gains from ParaNCE across both backbones. All values are in percentage points (pp).

MetricWithin-backbone comparisonQwen2.5-Omni gainAF3 gainGap (Qwen − AF3)
OverallDense SFT − Base+3.60+0.862.74
ParaNCE − Dense SFT+1.72+1.700.02
PerceptionDense SFT − Base+4.61+1.672.94
ParaNCE − Dense SFT+3.10+2.950.15
ReasoningDense SFT − Base+2.520.002.52
ParaNCE − Dense SFT+0.25+0.37−0.12

The cross-backbone difference in total adaptation gains arises primarily from Dense SFT rather than ParaNCE. On Overall, Dense SFT improves Qwen2.5-Omni by 3.60 pp versus 0.86 pp on AF3, creating a 2.74 pp gap that accounts for nearly all of the 2.76 pp difference in total gains. A similar pattern appears in Perception, where Dense SFT contributes a 2.94 pp gap (4.61 vs. 1.67 pp), compared with a 3.09 pp difference in total gains. Reasoning follows the same pattern, with Dense SFT gains of +2.52 pp on Qwen2.5-Omni and 0.00 pp on AF3.

In contrast, ParaNCE provides nearly identical additional gains over Dense SFT on Overall (+1.72 vs. +1.70 pp) and Perception (+3.10 vs. +2.95 pp), with statistically significant improvements on both backbones (all p < 0.01). Its smaller Reasoning gains are also comparable (+0.25 vs. +0.37 pp). These results indicate that the smaller total gains on AF3 largely reflect differences in Dense SFT improvements, while supporting the consistency of ParaNCE's additional benefits across backbones.

We hope these clarifications and supplementary experiments address the reviewer’s concerns about reasoning gains and cross-model consistency. Together, the results support the effectiveness of our focused paralinguistic supervision and demonstrate comparable additional benefits from ParaNCE across the two backbones.

References

[1] Qwen2.5-Omni Technical Report, arXiv, 2025.

[2] Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models, NeurIPS, 2025.

AuthorsOpenReview ↗

Replies to Reviewer 7991 – Review 2

Summarized Review 2: Performance Gap on Emotion Tasks

We thank the reviewer for raising W1 and Q1, which we summarize as asking why InfoNCE consistently outperforms ParaNCE across all five emotion-recognition benchmarks, particularly on EMO-DB, and whether ParaNCE’s contrastive objective nevertheless provides additional benefits for broader paralinguistic understanding. To address these concerns, we conducted a sampling ablation and extended our evaluation to paralinguistic tasks beyond emotion recognition.

InfoNCE benefits from our paired sampling

Our paper's InfoNCE baseline uses ParaNCE's hard-pair sampling, although InfoNCE does not require it [1]. Under this strategy, a contrastive pool of size 2k2k contains kk pairs with identical transcripts and speaker identities but different paralinguistic attributes. To assess the sampling benefit, we trained InfoNCE w/o Pairing with random negatives. The table reports accuracy (%), with parentheses indicating percentage-point changes from InfoNCE + Pairing.

MethodCREMA-DIEMOCAPESD-ENEMO-DBRAVDESS
Base27.7263.5229.9347.6621.25
Dense SFT31.9867.0837.3854.0233.26
InfoNCE + Pairing45.5469.5651.8169.5344.51
InfoNCE w/o Pairing29.20 (−16.34)66.26 (−3.30)42.58 (−9.23)53.64 (−15.89)32.36 (−12.15)
ParaNCE40.0966.7639.8750.6534.17

The ablation shows that our hard-pair sampling provides substantial and unexpected benefits for emotion recognition, as removing pairing reduces InfoNCE accuracy by 3.30–16.34 points across all five datasets. More broadly, all four adaptations improve acoustic perception and achieve significant mean emotion-recognition gains over Base (+6.73–18.17 points; two-sided paired t-tests, n=5n=5, all unadjusted p<0.05p<0.05), including an 8.29-point gain for ParaNCE (p=0.0188p=0.0188). These findings support the transfer from learning basic acoustic attributes to higher-level emotion understanding. The differences across training objectives further suggest that this transfer may depend on how each objective shapes acoustic representations, which we are continuing to investigate. See more details in Part 2 of our response to reviewer WPxz.

Broader Benefits of Task Transferring

To further establish ParaNCE's advantages in paralinguistic tasks transferring, we extended our evaluation on MMSU [2] and MSPB [3]. The table reports overall accuracy (%).

DatasetTaskNBaseInfoNCE + PairingInfoNCE w/o PairingParaNCE
MMSUSarcasm detection11671.5573.2868.9771.55
Intonation-based reasoning7870.5170.5174.3670.51
Prolonged-sound reasoning8495.2496.4396.4397.62
Pause-based reasoning9052.2258.8956.6762.22
Stress-based reasoning11043.6443.6442.7345.45
Emotion recognition11735.9053.8542.7454.70
Emotional-context reasoning11020.9127.2723.6430.00
Overall70553.6258.8755.7460.00
MSPBStatement/question intonation4057.5075.0062.5077.50
Ambiguity resolution4060.0057.5060.0060.00
Prosodic focus2429.1729.1729.1729.17
Focus operators1450.0071.4357.1471.43
Scalar meaning1050.0050.0060.0050.00
Irony comprehension1080.0070.0070.0070.00
Emotion recognition without context3246.8850.0046.8859.38
Emotion recognition with context837.5037.5037.5050.00
Overall17851.6956.7453.3760.11

Results show that ParaNCE achieves the highest overall accuracy on both benchmarks and leads or ties on 11 of the 15 tasks, indicating that our proposed ParaNCE benefits transfer beyond the supervised acoustic tasks to broader paralinguistic understanding.

Together, these findings highlight the substantial contribution of our proposed method. The unexpected benefit on InfoNCE also motivates further investigation into how sampling strategies and training objectives jointly shape paralinguistic learning and transfer. Meanwhile, the extended evaluations provide further evidence that ParaNCE's benefits transfer to broader paralinguistic tasks. We hope these findings adequately address concerns raised by the reviewer.

References

[1] Representation Learning with Contrastive Predictive Coding, arXiv preprint arXiv:1807.03748, 2018.

[2] MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark, ICLR, 2026.

[3] Can AI Understand Mandarin Speech Prosody? A Framework and Benchmark Showcase, Interspeech, 2025.

Reviewer_wPxzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether large speech models can understand "how people speak," rather than just "what they say." The authors constructed the PARAtaxis dataset, which includes four acoustic features: speaking rate, energy, pitch, and intonation, and found that existing large speech models have difficulty distinguishing different ways of speaking the same sentence. To address this, the authors propose ParaNCE, which uses specially designed difficult negative samples for contrastive learning to improve the model's ability to distinguish these sound features.

Critical strengths
  1. Constructed a valuable dataset for paralinguistic comparison. The authors constructed the PARAtaxis dataset, which keeps the spoken content and speaker identity constant while using acoustic measurements to verify variations in speaking rate, energy, pitch, and intonation. Comprising seven task formats, 11,272 training samples, and 500 diagnostic triplets, the dataset serves as a valuable resource for investigating how Speech LLMs understand paralinguistic information.
Critical weaknesses
  1. The dataset does not sufficiently isolate the target acoustic features, which may affect the training and diagnostic results of ParaNCE. The hard-negative pairs constructed by the authors only require that the target acoustic features be different, while the other three acoustic features can vary simultaneously. In addition, the caption for each audio segment contains information on all four acoustic properties. This leaves it unclear whether ParaNCE truly learned the acoustic differences specified by the authors, and whether diagnostic experiments can accurately assess the ability to distinguish specific acoustic features.

  2. The transfer relationship from the learning of basic acoustic features to high-level emotion understanding lacks sufficient validation. The authors argue that learning fundamental acoustic features helps models grasp higher-level emotional information. However, ParaNCE outperforms InfoNCE on MMSU Perception, especially on tasks directly related to training attributes, but lags behind InfoNCE on all five emotion recognition datasets. Furthermore, existing experiments have not sufficiently demonstrated that the gains in emotion recognition accuracy actually come from learning the four basic acoustic features, rather than from general additional speech task training. This also makes the relationship between basic acoustic discrimination ability and higher-level emotional understanding still unclear.

Questions for authors
  1. Does the model truly rely on the specified acoustic features to perform the discrimination? ParaNCE constructs hard negatives using speech pairs of the same text and the same speaker, but only requires that the target acoustic properties differ; other properties may change simultaneously, and the caption includes all four acoustic properties. Could the authors clarify how many of these speech pairs exhibit differences in multiple acoustic attributes simultaneously? Furthermore, would it be possible to re-evaluate the model using only those speech pairs where the other three attributes remain constant, in order to confirm that the performance gains from ParaNCE are indeed derived from the discrimination of the target acoustic attribute?

  2. Why did stronger basic acoustic perception abilities not lead to better performance in emotion recognition? The paper believes that learning four basic acoustic features can contribute to more advanced emotional understanding. However, ParaNCE outperforms InfoNCE on MMSU Perception, but lags behind InfoNCE on all five emotion recognition datasets. How does the author explain this difference?

  3. Is the performance improvement of ParaNCE stable under different random seeds?

7 comments
AuthorsOpenReview ↗

Replies to Reviewer wPxz – Review 1

Summarized Review 1: Isolation of Target Acoustic Attributes

We thank the reviewer for raising W1 and Q1, which we summarize as a shared concern about whether ParaNCE learns to discriminate individual acoustic attributes when multiple attributes may vary simultaneously and captions describe all four. We agree that performance on multi-attribute pairs alone cannot establish attribute-specific discrimination. Nevertheless, these pairs provide supervision for speech–caption alignment through differences in vocal delivery. To clarify the contribution of individual attributes, we examine how frequently multiple attributes differ within speech pairs and whether ParaNCE’s gains persist when the other three attributes are held constant, using dataset statistics and a controlled subset evaluation, respectively.

How frequently do multiple acoustic attributes differ?

We consider pairs differing in only one acoustic attribute while preserving the other three ideal for this evaluation. However, current controllable TTS systems cannot reliably achieve such precise control, as also documented in InstructTTSEval and E-VOC. Our construction therefore involved repeated synthesis and acoustic screening, totaling 6,110,501 candidate evaluations, including 6,020,179 with acoustic features. The final dataset contains 5,636 training pairs (11,272 examples), 500 diagnostic triplets, and 1,488 instruction-tuning examples. To assess the extent of attribute variation, we analyzed the number of differing acoustic attribute labels (1 to 4) in PARAtaxis-Train and PARAtaxis-Diag, separately for speaking rate, energy, pitch, and intonation. All pairs share the same transcript and speaker.

PARAtaxis-Train

Target attribute1 differs2 differ3 differ4 differTotal
Speaking rate218715380951,408
Energy2396204491071,415
Pitch363666330461,405
Intonation399634307681,408
Total1,2192,6351,4663165,636

PARAtaxis-Diag

Target attribute1 differs2 differ3 differ4 differTotal
Speaking rate1470374125
Energy1657439125
Pitch2661344125
Intonation3255326125
Total8824314623500

Among these, 1,219 training pairs and 88 diagnostic pairs differ in only one attribute label. We use the 88 diagnostic pairs, where the other three attribute labels remain constant, for the following re-evaluation.

Do ParaNCE’s gains persist when only the target attribute label differs?

To address the second aspect, we re-evaluated Base, Dense SFT, InfoNCE, and ParaNCE on the 704 matching questions derived from pairs differing in only one attribute label, using the same protocol as the paper’s pilot experiments. This tests whether ParaNCE’s gains persist when the other three attribute labels remain constant. The table reports Qwen2.5-Omni-7B Overall accuracy (%) across this subset and the remaining diagnostic subsets, covering all 4,000 matching questions with equal numbers of speech-to-caption and caption-to-speech questions.

Differing attribute labelsQuestionsBaseDense SFTInfoNCEParaNCEParaNCE−InfoNCE (pp)
1: target only70450.5755.9756.8263.35+6.53
21,94452.0157.9263.2270.06+6.84
31,16851.9758.9063.1070.29+7.19
418453.2659.7871.7478.80+7.07

First, increasing the number of differing attributes makes the two deliveries distinguishable along more acoustic dimensions, providing additional cues for matching speech with its description. Both contrastive models show higher accuracy as the number of differing attributes increases, while Base remains near chance and Dense SFT shows smaller improvements.

Second, ParaNCE consistently outperforms InfoNCE in both single-attribute and multi-attribute contrasts. Notably, its advantage persists when the other three attribute labels remain constant, supporting its ability to discriminate individual acoustic attributes rather than relying solely on combined differences.

Together, these analyses address W1 and Q1 by quantifying simultaneous attribute variation and examining its influence on discrimination performance. Although multi-attribute contrasts provide additional acoustic cues, ParaNCE’s advantage persists when only the target attribute label differs. This provides more direct evidence of target-attribute discrimination and shows that ParaNCE’s gains do not rely solely on combined acoustic differences.

AuthorsOpenReview ↗

Replies to Reviewer wPxz – Review 2

Summarized Review 2: Transfer from Acoustic Perception to Emotion Understanding

We thank the reviewer for raising W2 and Q2 regarding whether emotion-recognition gains come from learning basic acoustic distinctions rather than additional speech-task training, and why ParaNCE's stronger acoustic perception does not yield better emotion recognition than InfoNCE. Our adaptation introduces no conventional speech-task training. All training data and supervision, including InstFT, come exclusively from PARAtaxis, constructed around speaking rate, energy, pitch, and intonation. We will clarify this in the revised paper.

InfoNCE benefits from our paired sampling

Our paper's InfoNCE baseline uses ParaNCE's hard-pair sampling, although InfoNCE does not require it [1]. Under this strategy, a contrastive pool of size 2k2k contains kk pairs with identical transcripts and speaker identities but different paralinguistic attributes. To assess the sampling benefit, we trained InfoNCE w/o Pairing with random negatives. The table reports accuracy (%), with parentheses indicating percentage-point changes from InfoNCE + Pairing.

MethodCREMA-DIEMOCAPESD-ENEMO-DBRAVDESS
Base27.7263.5229.9347.6621.25
Dense SFT31.9867.0837.3854.0233.26
InfoNCE + Pairing45.5469.5651.8169.5344.51
InfoNCE w/o Pairing29.20 (−16.34)66.26 (−3.30)42.58 (−9.23)53.64 (−15.89)32.36 (−12.15)
ParaNCE40.0966.7639.8750.6534.17

The ablation reveals a substantial contribution from paired sampling. Removing pairing reduces InfoNCE accuracy by 3.30–16.34 points across all five datasets, indicating that our hard-pair sampling strategy provides substantial and unexpected benefits for emotion recognition.

To further validate the consistent improvements in acoustic perception and emotion recognition, we test mean improvements over Base across the five datasets. The table reports acoustic and mean emotion accuracy (%) with two-sided paired t-tests of emotion gains over Base across five datasets (n=5n=5).

MethodMMSU overallMMSU perceptionFour tasksDiagnosisEmotion meanEmotion gainp-value
Base59.1646.5926.0351.8038.020–
Dense SFT63.0049.7731.0557.9544.75+6.730.010832
InfoNCE + Pairing62.7049.4228.3162.4556.19+18.170.004571
InfoNCE w/o Pairing62.2648.4929.6858.5544.81+6.790.037542
ParaNCE64.7252.8736.9969.3546.31+8.290.018833

All four adaptations improve acoustic perception and significantly improve mean emotion accuracy over Base (all unadjusted p<0.05). These consistent improvements provide further evidence that learning the four basic acoustic attributes can support higher-level emotion understanding. The differences across training objectives suggest that this transfer may depend on how each objective shapes the learned acoustic representations. We are continuing to investigate the mechanisms underlying these differences.

Broader paralinguistic transfer beyond emotion recognition

To further establish ParaNCE's advantages in broader paralinguistic understanding, we extended our evaluation beyond the four acoustic attributes and emotion recognition. We compared ParaNCE with both InfoNCE variants on 15 paralinguistic tasks comprising 883 examples from MMSU [2] and MSPB [3], covering sarcasm, prosodic reasoning, ambiguity, focus, irony, and emotion. The table reports overall accuracy (%).

MethodMMSU subsetMSPB
Base53.6251.69
InfoNCE + Pairing58.8756.74
InfoNCE w/o Pairing55.7453.37
ParaNCE60.0060.11

Results show that ParaNCE achieves the highest overall accuracy on both benchmarks and leads on 11 of the 15 tasks, indicating that our proposed ParaNCE benefits extend beyond the four supervised acoustic attributes to broader paralinguistic understanding.

We believe these findings provide stronger evidence that learning foundational acoustic attributes can support higher-level paralinguistic understanding, while highlighting how training objectives shape this transfer. We are continuing to investigate the underlying mechanisms and hope this encourages further research into speech LLM understanding. We hope these additional results adequately address the reviewer's concerns.

References

[1] Representation Learning with Contrastive Predictive Coding, arXiv, 2018.

[2] MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark, ICLR, 2026.

[3] Can AI Understand Mandarin Speech Prosody? A Framework and Benchmark Showcase, Interspeech, 2025.

AuthorsOpenReview ↗

Replies to Reviewer wPxz – Review 3

Summarized Review 3: Robustness Across Random Seeds

We thank the reviewer for raising Q3 regarding the consistency of ParaNCE’s improvements across random seeds. To assess this, we randomly selected five seeds from 0–50 (11, 23, 25, 30, and 42) and trained and evaluated Dense SFT, InfoNCE, and ParaNCE under identical settings.

Robustness Across Random Seeds

The resulting MMSU accuracies (%) are summarized below:

MethodOverallPerceptionReasoning
Dense SFT62.47 ± 0.9049.43 ± 1.1576.36 ± 0.76
InfoNCE62.72 ± 0.1849.78 ± 0.6476.51 ± 0.56
ParaNCE64.52 ± 0.3253.12 ± 0.4476.67 ± 0.69

Across the five-seed evaluation, ParaNCE’s gains over Dense SFT and InfoNCE remain statistically significant for both Overall (p = 0.002498 and 0.000461, respectively) and Perception (p = 0.001837 and 0.000213, respectively). Meanwhile, mean Reasoning accuracy shows no decrease relative to either baseline, showing that the perceptual gains do not come at the expense of paralinguistic reasoning.

We hope these additional results address the reviewer’s concern and provide clearer evidence of the robustness of ParaNCE’s improvements across random seeds.

Reviewer_fQEiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the gap between the lexical and paralinguistic understanding of Speech LLMs. It aims to address the issue that Speech LLMs are more dependent on the lexical content of the input. It first proposes PARAtaxis, a synthetic speech corpus with lexical and prosodic contrasting samples covering four dimensions: speaking rate, energy, pitch, and intonation. The corpus is used to diagnose the linguistic-paralinguistic asymmetry and the representation level asymmetry existing in the baseline speech LLMs. It further proposes a ParaNCE loss which aims to close the lexical-acoustic gap identified in the diagnoses. Experiments on MMSU and five emotion datasets show improved perception and emotion understanding post ParaNCE guided training over the proposed dataset, although the InfoNCE achieves higher emotion-recognition accuracy across all five datasets.

Critical strengths
  1. The preliminary experiments in section 6 establish the gap and strengthen the motivation of the work.
  2. The dataset paired with the ParaNCE guided training helps the speech llms improve their performance on MMSU and five emotion datasets.
  3. Additional experiments on the PARAtaxis-diag show that finetuning on the proposed dataset also helps the model to improve the quantitative metric used in the preliminary experiments.
Critical weaknesses
  1. Limited effectiveness of ParaNCE – While the ParaNCE guided training achieves meaningful and substantial improvement on MMSU and emotion datasets as reported in Table 3 and 4, the gap between ParaNCE and InfoNCE is interesting. The InfoNCE consistently overshadows ParaNCE over all the 5-emotion dataset, especially by around 24% on EMO-DB after instruction fine-tuning.
  2. Difference from VoxParadox and ContraProST – While the authors discuss the two related works in 2.1 and 2.2, briefly discussing how the PARAtaxis corpus is different from VoxParadox, and ContraProST, it would be better to discuss this in more details, since these corpus aim to solve the same underlying problem raised in the work. Specifically, in addition to discussing how the proposed dataset is different from these two existing ones, it should also be established how these differences add value to the solution. Furthermore, comparing models trained on these existing datasets against models trained on PARAtaxis, where the supervision formats permit a fair comparison, would help strengthen the value of the dataset.
Questions for authors
  1. With reference to (1) in weakness, does the paralinguistic pairing of ParaNCE add enough advantages that translates to broader paralinguistic understanding?
  2. Does the paralinguistic supervision affect the model's original capabilities, e.g. speech recognition, instruction-following etc.?
7 comments
AuthorsOpenReview ↗

Replies to Reviewer fQEi – Review 1

Summarized Review 1: Broader Benefits of Paralinguistic Learning

We thank the reviewer for raising W1 and Q1, which we summarize as concerns about ParaNCE’s broader benefits for paralinguistic understanding, given that InfoNCE consistently outperforms ParaNCE across all five emotion-recognition datasets, particularly on EMO-DB. To investigate these questions, we conducted a sampling ablation and extended our evaluation to broader paralinguistic tasks beyond emotion recognition.

InfoNCE benefits from our paired sampling

Our paper's InfoNCE baseline uses ParaNCE's hard-pair sampling, although InfoNCE does not require it [1]. Under this strategy, a contrastive pool of size 2k2k contains kk pairs with identical transcripts and speaker identities but different paralinguistic attributes. To assess the sampling benefit, we trained InfoNCE w/o Pairing with random negatives. The table reports accuracy (%), with parentheses indicating percentage-point changes from InfoNCE + Pairing.

MethodCREMA-DIEMOCAPESD-ENEMO-DBRAVDESS
Base27.7263.5229.9347.6621.25
Dense SFT31.9867.0837.3854.0233.26
InfoNCE + Pairing45.5469.5651.8169.5344.51
InfoNCE w/o Pairing29.20 (−16.34)66.26 (−3.30)42.58 (−9.23)53.64 (−15.89)32.36 (−12.15)
ParaNCE40.0966.7639.8750.6534.17

The ablation shows that our hard-pair sampling provides substantial and unexpected benefits for emotion recognition, as removing pairing reduces InfoNCE accuracy by 3.30–16.34 points across all five datasets. More broadly, all four adaptations improve acoustic perception and achieve significant mean emotion-recognition gains over Base (+6.73–18.17 points; two-sided paired t-tests, n=5n=5, all unadjusted p<0.05p<0.05), including an 8.29-point gain for ParaNCE (p=0.0188p=0.0188). These findings support the transfer from learning basic acoustic attributes to higher-level emotion understanding. The differences across training objectives further suggest that this transfer may depend on how each objective shapes acoustic representations, which we are continuing to investigate. See more details in Part 2 of our response to reviewer WPxz.

Broader Benefits of Paralinguistic Learning

To further establish ParaNCE's advantages in broader paralinguistic understanding, we extended our evaluation on MMSU [2] and MSPB [3]. The table reports overall accuracy (%).

DatasetTaskNBaseInfoNCE + PairingInfoNCE w/o PairingParaNCE
MMSUSarcasm detection11671.5573.2868.9771.55
Intonation-based reasoning7870.5170.5174.3670.51
Prolonged-sound reasoning8495.2496.4396.4397.62
Pause-based reasoning9052.2258.8956.6762.22
Stress-based reasoning11043.6443.6442.7345.45
Emotion recognition11735.9053.8542.7454.70
Emotional-context reasoning11020.9127.2723.6430.00
Overall70553.6258.8755.7460.00
MSPBStatement/question intonation4057.5075.0062.5077.50
Ambiguity resolution4060.0057.5060.0060.00
Prosodic focus2429.1729.1729.1729.17
Focus operators1450.0071.4357.1471.43
Scalar meaning1050.0050.0060.0050.00
Irony comprehension1080.0070.0070.0070.00
Emotion recognition without context3246.8850.0046.8859.38
Emotion recognition with context837.5037.5037.5050.00
Overall17851.6956.7453.3760.11

Results show that ParaNCE achieves the highest overall accuracy on both benchmarks and leads or ties on 11 of the 15 tasks, indicating that our proposed ParaNCE benefits extend beyond the four supervised acoustic attributes to broader paralinguistic understanding.

Together, these findings highlight the substantial contribution of our paralinguistic paired sampling. The unexpected benefit on InfoNCE also motivates further investigation into how sampling strategies and training objectives jointly shape paralinguistic learning and transfer. Meanwhile, the extended evaluations provide further evidence that ParaNCE's benefits generalize to broader paralinguistic understanding. We hope these findings address both concerns raised by the reviewer.

References

[1] Representation Learning with Contrastive Predictive Coding, arXiv preprint arXiv:1807.03748, 2018

[2] MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark, ICLR, 2026

[3] Can AI Understand Mandarin Speech Prosody? A Framework and Benchmark Showcase, Interspeech, 2025

AuthorsOpenReview ↗

Replies to Reviewer fQEi – Review 2, Part I

Summarized Review 2: Dataset Design and Comparisons with VoxParadox and ContraProST

We summarize the reviewer’s W2 as raising three connected questions: (1) how our research question relates to those investigated by VoxParadox [1] and ContraProST [2]; (2) how PARAtaxis differs and why these differences matter for diagnosing and improving paralinguistic understanding; and (3) what empirical comparisons can establish about its utility as a training resource.

1. Relationship to the research questions in VoxParadox and ContraProST

Thank you for encouraging a fuller discussion of these closely related studies and we agree that Sections 2.1 and 2.2 currently discuss them too briefly.

VoxParadox. VoxParadox investigates why Speech LLMs fail to use acoustic evidence when it conflicts with linguistic content. Their layer-wise probes identify both degradation of paralinguistic information and failures to use attributes that remain decodable. These findings motivate prompt-conditioned encoder-layer mixing and preference optimization, addressing how relevant evidence reaches the language model and influences its responses. This provides an important account of the gap between information availability and utilization.

ContraProST. ContraProST introduces a double-contrastive benchmark to evaluate whether speech-to-text translation systems use prosody to produce translations consistent with the intended meaning. Each example pairs two realizations of the same utterance with corresponding translations, covering stress, breaks, intonation, emotion, and politeness. By evaluating both conditional likelihoods and generated translations, they show that sensitivity to prosody in model scores does not necessarily translate into appropriate translation outputs. ContraProST thus provides a targeted evaluation of prosodic understanding without introducing a training method.

We will expand Sections 2.1 and 2.2 accordingly, giving fuller consideration to the research questions and findings of both studies.

2. Dataset differences and their role in diagnosis and training

The key distinction lies in how each dataset uses controlled contrasts. While VoxParadox and ContraProST primarily use them to evaluate specific aspects of speech understanding, PARAtaxis is designed to support both the diagnosis and improvement of paralinguistic discrimination. This motivates three designs: transcript- and speaker-matched hard negatives, acoustically grounded labeling, and multiple task formats. We compare the datasets below before discussing the role of each design choice.

AspectVoxParadoxContraProSTPARAtaxis (ours)
MotivationAcoustic evidence under lexical conflictProsody-aware translationParalinguistic diagnosis and learning
ContrastSpoken claims vs. acoustic cuesProsodic variants and translation meaningsSame transcript/speaker, different delivery
StructureAudio + 4-option QAPaired speech and translationsSpeech–caption matches/mismatches
TasksAttribute QA; acoustic ranking; speaker ID/countTranslation scoring/generationCaptioning; explanation; comparison; sequence; localization; retrieval; correction
Speech propertiesAge, gender, emotion, pitch, volume, speed, vocal range, intonation, speakerStress, breaks, intonation, emotion, politenessRate, energy, pitch, intonation
Acoustic verificationASR/classifier checksLLM interpretations; F0/intensity/duration; ASR; classifiersWaveform measurements + unified labeling
Size2,000 examples1,311 En–De; 1,294 En–Es; 1,386 En–Ja12,760 training; 500 diagnostic triplets

These differences contribute to our approach in three specific ways.

  • Paralinguistic Hard Negatives. By matching transcripts and speakers while varying vocal delivery, PARAtaxis removes lexical and speaker shortcuts in speech–text matching. These contrasts directly test whether models distinguish paralinguistic differences and provide targeted supervision to improve this ability.

  • Acoustic Verification. We measure speaking rate, energy, pitch, and intonation from synthesized audio and assign labels using consistent criteria. This grounds hard negatives in measurable acoustic differences, ensures reproducible supervision, and enables scalable data construction without manual annotation.

  • Multiple Task Formats for Paralinguistic Learning. Seven task formats require models to recognize, describe, compare, locate, and reason about the same four acoustic attributes. This provides diverse supervision for developing foundational paralinguistic skills and applying them across different speech-understanding tasks.

We hope this clarifies how the dataset differences support both the diagnosis of paralinguistic discrimination and the construction of explicit learning targets. We will revise the related-work discussion to make these design choices and their practical value clearer.

AuthorsOpenReview ↗

Replies to Reviewer fQEi – Review 2, Part II

3. Empirical comparisons using the released datasets and model

Having discussed the relationships and dataset differences among PARAtaxis, VoxParadox, and ContraProST, we now examine their empirical utility. Following the reviewer’s suggestion, we compare the three datasets under the same Dense SFT procedure and backbone. We additionally compare the released VoxParadox PCLM+DPO model with our ParaNCE-adapted model, extending beyond the requested SFT comparison. We first summarize the original studies’ architectures, training data, methods, and objectives.

AspectVoxParadoxContraProSTPARAtaxis / ParaNCE (ours)
ArchitectureAF3, Qwen2-Audio + PCLMSeamlessM4T, SALMONN, Whisper–NLLBQwen2.5-Omni, AF3
Training dataAudio QA, paralinguistic tasks, DPO pairsN/AMatched speech–caption pairs; 4 cues, 7 tasks
Training methodPCLM SFT + DPON/AJoint NTP + ParaNCE
ObjectiveAcoustic reliance under lexical conflictProsody-sensitive translationParalinguistic diagnosis and improvement

Training comparison on the released resources.

For this comparison, we use the released VoxParadox and ContraProST benchmarks and PARAtaxis as SFT training data, retaining their original labels. All models are initialized from the same Qwen2.5-Omni-7B backbone and trained for one epoch with identical Dense SFT hyperparameters. We evaluate them on the same MMSU benckmark using the training and inference settings identical with our paper, accuracy (%) is reported.

Training datasetOverallPerceptionReasoningEmotionIntonationPitchSpeedVolumeStressPauseProlonged soundVocal rangeAgeGenderSpeaker countEmotional-context reasoningIntonation-basedPause-based
Base59.4045.1674.5935.9019.8232.4119.2732.7343.6428.0457.8330.5651.9265.8344.1720.9166.6752.22
VoxParadox42.6233.9551.8638.4636.9430.5620.1824.5530.0020.5653.0123.1527.8886.6720.0032.7355.1334.44
ContraProST60.2246.2875.0837.6117.1228.7020.1830.0064.5533.6461.4531.4848.0867.5044.1727.2773.0855.56
PARAtaxis63.0049.7777.1153.8529.7334.2627.5232.7349.0945.7975.9030.5655.7785.0038.3329.0974.3661.11

PARAtaxis achieves the highest accuracy across all three metrics: 63.00% Overall, 49.77% Perception, and 77.11% Reasoning, and leads on nine of the 15 subtasks. VoxParadox and ContraProST show relative strengths on certain subtasks, consistent with their respective emphasis on gender and intonation [1] and sentence stress [2]. Overall, these results support the practical value of PARAtaxis as a training resource for speech-understanding tasks.

Comparison with the Released VoxParadox PCLM+DPO Model.

Beyond the dataset-level Dense SFT comparison, we further compare our PARAtaxis/ParaNCE adaptation with the publicly released VoxParadox PCLM+DPO model weight, both based on the same pretrained Audio Flamingo 3 (AF3) backbone. We evaluate these models alongside pretrained AF3 and Dense SFT on MMSU, reporting accuracy (%) under our evaluation protocol.

AF3 adaptationOverallPerceptionReasoning
Pretrained AF359.4643.9575.99
Dense SFT60.3245.6275.99
VoxParadox: PCLM+DPO60.1846.5574.71
ParaNCE61.1647.0976.16

Our ParaNCE-adapted AF3 achieves the highest accuracy across all three MMSU metrics, further supporting the effectiveness of PARAtaxis and ParaNCE. Notably, our adaptation uses only 11,272 examples (15.3 hours of audio) for one epoch, requiring 1.61 H100 GPU-hours for training. In comparison, VoxParadox reports approximately 1.267 million supervised examples and 20,000 DPO preference pairs. Despite differences in training setups, these results demonstrate strong performance with a compact dataset and modest training cost.

Together, we hope that the expanded related-work discussion and additional experiments help address the reviewer's concern.

References

[1] Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox, ICML, 2026

[2] Speech Is More than Words: Do Speech-to-Text Translation Systems Leverage Prosody?, WMT, 2024

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to Reviewers fQEi, xPxz, and 7991

We sincerely thank reviewers fQEi, xPxz, and 7991 for their constructive feedback and for recognizing the well-motivated problem formulation, the linguistic–paralinguistic gap established by our preliminary analyses, and the value of PARAtaxis as a research resource. We also appreciate their acknowledgment of the improvements in speech perception, emotion understanding, and paralinguistic discrimination, particularly the benefits of learning foundational acoustic cues for tasks beyond those directly supervised during training.

Below, we consolidate the weaknesses and questions raised across the three reviews into thematic summaries reflecting our understanding of the reviewers’ main concerns, and address each in turn.

AuthorsOpenReview ↗

Meta-Review: Summary of Concerns and Responses

We thank reviewers fQEi, wPxz, and 7991 for their careful assessment and constructive discussion, among whom reviewer fQEi considers the questions largely addressed. Below, we summarize the concerns and evidence from the full rebuttal.

  1. Dataset validity and attribute isolation (wPxz). We quantified simultaneous attribute variation in training and diagnostic sets and re-evaluated 88 diagnostic pairs differing only in the target attribute label. ParaNCE outperforms InfoNCE across single- and multi-attribute subsets, supporting attribute-specific discrimination beyond reliance on combined differences.

  2. Positioning, alternative methods, and cross-backbone gains (fQEi, 7991). We clarified how PARAtaxis supports diagnosis and training through transcript- and speaker-matched contrasts, acoustic verification, and multiple task formats. With the same backbone and Dense SFT setup, PARAtaxis outperforms the released VoxParadox and ContraProST resources on MMSU. Under our MMSU protocol, ParaNCE outperforms VoxParadox’s released PCLM+DPO checkpoint on MMSU despite its smaller training budget. AF3 tuning reveals comparable gains over Dense SFT across both backbones; total-gain differences largely reflect Dense SFT gains. We related the modest reasoning gains to supervision focused on acoustic perception rather than semantic and contextual inference.

  3. Objective differences, emotion transfer, and robustness (all reviewers). We first clarified that the original adaptation, including InstFT, uses only PARAtaxis supervision on speaking rate, energy, pitch, and intonation, without conventional speech-task or emotion-category training. The sampling ablation showed substantial emotion benefits from pairing, while all adaptations still improved mean emotion accuracy over Base. Reviewer wPxz then noted that pairing alone did not explain the gap between paired methods and requested clearer loss definitions. We therefore clarified the objectives and paired control, examined the original batch/update differences, and added matched-setting three-seed comparisons. ParaNCE outperforms the paired InfoNCE control on MMSU and all five emotion datasets with statistical significance. Results also reveal a schedule-dependent tradeoff: smaller batches with more updates favor perception, whereas larger batches with fewer updates favor emotion transfer. Additionally, a five-seed experiment was conducted to verifie the robustness of ParaNCE’s paralinguistic gains.

  4. Generalization beyond basic attributes (fQEi, 7991). Our initial MMSU/MSPB comparisons against both InfoNCE variants showed ParaNCE leading aggregate accuracy on both subsets. Evaluations on MUStARD, ADEPT, MSPB, and MMSU showed broader transfer without task-specific training, including gains in sentence interpretation, speaker attitudes, contextual sarcasm, and emotion understanding. Reviewer 7991 then requested clearer separation of tasks overlapping with training attributes from complex semantic or pragmatic tasks, and evidence of benefits for subsequent learning. We then reorganized the evaluations and added fine-tuning on MUStARD and StanceBench. With identical training settings and 120 labeled examples per task, ParaNCE initialization yields higher final accuracy than Base initialization, suggesting potential benefits for subsequent learning of complex paralinguistic tasks.

  5. General-capability retention (fQEi, 7991). We first evaluated ten benchmarks covering speech recognition, intent understanding, and instruction following. These showed generally modest declines; the original PARAtaxis-only InstFT recovered some performance without general-task training. Reviewer 7991 highlighted the remaining declines and suggested joint training with a matched Dense SFT baseline. We evaluated the suggested joint training before and after InstFT, including general-data replay. Joint ParaNCE matches or exceeds Joint Dense SFT on eight of ten general benchmarks per setting, retaining its MMSU advantage. Compared with the corresponding original ParaNCE models, it improves six to seven general benchmarks at an MMSU cost of 0.28–0.56 percentage points, reducing tradeoffs.

The current revision focuses on presentation, including expanded related work and clearer exposition. After completing the remaining multi-seed emotion experiments, we plan to consolidate the empirical updates, including clearer InfoNCE ablation definitions and an appendix analysis of how batch size and update frequency affect the balance between acoustic perception and emotion transfer.

We particularly welcome Reviewer wPxz’s feedback on the clarified objectives and updated emotion analyses, and Reviewer 7991’s feedback on the additional generalization and joint-training experiments. Their feedback would help integrate these findings into a fuller account of broader paralinguistic transfer and general-capability retention in the next revision.

AuthorsOpenReview ↗

Summary of Core Contributions and New Insights

To support an overall assessment of our study, we summarize its core contributions and the further insights gained through the rebuttal. We introduced PARAtaxis to examine whether speech LLMs can distinguish how the same words are spoken. Nine pretrained speech LLMs readily distinguished changes in wording but struggled with different deliveries of the same words by the same speaker. This motivated ParaNCE, which explicitly matches each delivery to its own description of speaking rate, energy, pitch, and intonation. Our study showed that learning these distinctions improves acoustic perception and transfers to broader paralinguistic tasks, including higher-level emotion understanding. The additional evidence deepens this finding by clarifying which distinctions are learned, how the benefits generalize, and where the data and objective each add value.

FindingWhat our study initially establishedWhat the rebuttal adds
Discrimination of individual cuesParaNCE outperformed InfoNCE on the full diagnostic set, where the number of differing cue labels varies across pairs.ParaNCE retains a comparably large discrimination advantage even when only the target attribute label differs. [wPxz, Review 1]
Transfer beyond acoustic perceptionLearning the four cues benefited broader paralinguistic tasks, including emotion recognition without emotion supervision.The learned distinctions benefit tasks directly or indirectly related to the supervised cues, including sentence interpretation, speaker attitude, and sarcasm, and provide a stronger initialization for subsequent task learning. [7991, Review 4] · [7991, Reservation 1, Part II]
What hard-pair supervision contributesParaNCE led on perception, whereas InfoNCE + Pairing led on emotion recognition.Hard-pair sampling and explicit pair supervision make complementary contributions: pairing supports emotion transfer, while supervising both pair members adds reliable benefits in perception and emotion under matched training schedules. The number of updates partly contributes to the balance of transfer across tasks. [fQEi, Review 1] · [wPxz, follow-up II]
Reliability and breadth of ParaNCE's advantageParaNCE improved MMSU on both Qwen2.5-Omni and Audio Flamingo 3 (AF3), with different overall adaptation gains.ParaNCE's added perceptual value is consistent across the two backbones, and five-seed evaluations on Qwen establish statistically significant gains on MMSU and emotion recognition. Its advantage over both InfoNCE variants also extends to the evaluated broader MMSU and MSPB task groups. [7991, Review 1, Part II] · [wPxz, Review 3] · [wPxz, Review 2]
Value of PARAtaxis as a training resourcePARAtaxis supported diagnosis and training, with internal ablations establishing the benefit of ParaNCE on this resource.PARAtaxis has training value beyond the proposed objective, providing more effective paralinguistic supervision than the compared VoxParadox and ContraProST resources. Together, PARAtaxis and ParaNCE also offer an effective alternative to VoxParadox's released PCLM+DPO system. [fQEi, Review 2, Part II]
Compatibility with general speech capabilitiesOur study established paralinguistic gains using only PARAtaxis for adaptation, including instruction tuning.Joint supervision makes the specialization tradeoff more favorable: paralinguistic gains are largely retained while costs to speech recognition and intent understanding are reduced. ParaNCE retains its added value over jointly trained Dense SFT. [7991, Review 5] · [7991, Reservation 2, Part II]

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.