acceptodds
Under review as a conference paper at ICLR 2027

Told, Then Aware: On-Policy Distillation for Empathetic Speech-to-Speech Dialogue

Abstract

The same sentence spoken in a different tone calls for a different reply, yet end-to-end speech-to-speech (S2S) models usually respond to the words and largely ignore the voice, even when the delivery contradicts what is said. The common remedy is more empathetic dialogue data with paired annotations, but that data is scarce. We find the solution already latent in the model: a pretrained S2S model can carry emotion in its own speech, just not always the emotion empathy calls for, and it can answer empathetically both in content and in how it sounds as soon as the user's affect is explicitly stated in words. What is missing is not the capability but an objective that requires the voice to change the reply. We therefore adopt on-policy distillation (OPD): the teacher is prompted with the user's affect information as a training-time privilege, and this frozen privileged teacher provides dense per-position distributional supervision over both text and speech positions along the student's own sampled response trajectories. On SpeechParaling-Bench, our method on one backbone improves the Chinese score from 43.6 to 56.3 and the English score from 60.5 to 71.8, outperforming all evaluated systems. We observe consistent gains on EchoMind, VocalBench-zh, and with an alternative backbone. Furthermore, human evaluation confirms that the model trained via our approach achieves the best performance, demonstrating OPD’s potential for generating empathetic spoken responses.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
speech language models, speech-to-speech, empathetic spoken dialogue, paralinguistic-aware speech generation, on-policy distillation, privileged information

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1ihjOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an on-policy distillation (OPD) framework for improving empathetic speech-to-speech (S2S) dialogue. Motivated by the perception–behavior gap, where speech models may encode paralinguistic information without appropriately incorporating it into their responses, the authors use a frozen privileged teacher that receives the user's speech, transcript, and explicit emotion description. A student model receives only the speech input and learns from the teacher's token-level probability distributions over both textual and speech outputs through Jensen–Shannon divergence.

The method is trained using 34,305 synthetic Chinese and English utterances and evaluated on two S2S backbones across three benchmarks. The authors report improvements over the original backbones on several measures of response appropriateness and vocal expressiveness, alongside a small-scale human evaluation. They further provide linear probing experiments and theoretical arguments intended to explain why the proposed objective encourages acoustic emotion grounding.

Critical strengths
  1. The proposed training procedure improves the reported performance of two different S2S backbones across several automatic evaluation metrics, including vocal expressiveness and speech-response appropriateness. Incorporating speech-token-level supervision is also potentially useful for extending privileged distillation to end-to-end spoken response generation.
Critical weaknesses
  1. The proposed framework closely resembles ParaBridge, which already addresses the perception–behavior gap through privileged on-policy self-distillation, with student-generated trajectories and dense token-level distributional supervision. The main distinctions here appear to be explicitly providing emotion labels and transcripts to the teacher, freezing the teacher, and extending the objective to speech tokens. However, the paper lacks a controlled comparison against a ParaBridge-style approach using the same backbone, training data, and computational budget. Consequently, the incremental contribution is not convincingly established. More importantly, I find the central theoretical justification problematic. Equations (6)–(7) claim that the distillation objective isolates the contribution of emotion and reaches zero only when the student's audio-derived representation is sufficient for the user's affect. However, the teacher receives both the transcript and emotion description, while the student receives only audio. Furthermore, the teacher and student have different effective parameters because the student contains trainable adapters. Therefore, their distributional differences cannot be attributed exclusively to emotion. The objective may also be minimized through alternative shortcuts rather than genuine acoustic emotion grounding. This concern is particularly relevant because approximately 91.2% of the synthetic training examples deliberately pair positive lexical content with downcast delivery or negative lexical content with cheerful delivery. Such a systematic relationship could allow a model to learn an inverse lexical-valence heuristic without reliably attending to vocal emotion. The paper does not sufficiently rule out this possibility.

  2. I identified an important numerical inconsistency between Table 3 and Table 17. Table 3 reports an EchoMind C4 score of 4.1280 for SFT+PO and 4.3248 for OPD, implying a difference of +0.1968. However, Table 17 reports a paired mean difference of only +0.0768 for OPD versus SFT+PO. Similarly, Table 3 reports EmoAlign scores of 35.42 and 34.62, implying that OPD performs worse by 0.80, whereas Table 17 reports a positive difference of +0.6540 in favor of OPD. If these tables refer to the same checkpoints and evaluation items, the reported differences are mathematically inconsistent. The paper does not clearly explain any difference in the evaluated data, checkpoints, or aggregation procedures that could reconcile these numbers. This substantially undermines confidence in the reported statistical comparisons, particularly because Table 17 is intended to substantiate the effectiveness of OPD relative to alternative training objectives. Furthermore, the evaluation does not sufficiently establish that the improvements reflect genuine acoustic emotion understanding. Several automatic metrics assess the generated response or its emotional expression without directly observing the user's original vocal delivery. The human evaluation is also very limited, involving only 20 utterances and eight retained conflict examples. The empathy rating increases only from 3.85 to 3.88, without reported inter-annotator agreement, confidence intervals, or significance tests for the human comparisons.

Questions for authors
  1. Could the authors explain the discrepancies between Tables 3 and 17? Specifically, why is the OPD-versus-SFT+PO EchoMind C4 difference reported as +0.0768 in Table 17 when the corresponding Table 3 scores imply +0.1968? Likewise, why does the EmoAlign comparison change direction between the two tables? Were different checkpoints, evaluation subsets, or aggregation procedures used? Please identify the exact source of these differences and provide corrected or reconciled results.

  2. Could the authors provide a rigorous justification for Equation (7), explicitly accounting for the additional transcript available to the teacher and the differing effective teacher–student parameters? Under what assumptions does minimizing the proposed divergence necessarily require learning acoustic affect, rather than reproducing teacher behavior through lexical information or other shortcuts? In particular, how can this claim be reconciled with the strong inverse relationship between lexical and vocal valence in the training data?

  3. What is the specific methodological contribution beyond ParaBridge, and what experimental evidence isolates its effect? A controlled comparison using the same backbone, dataset, and training budget would materially affect my assessment. In addition, can the authors provide downstream results isolating emotion-only, transcript-only, and combined teacher privileges, as well as the independent contribution of speech-token supervision?

  4. Could the authors clarify how the human evaluators were recruited and report inter-annotator agreement, confidence intervals, and statistical comparisons between OPD and its original backbone? Given the small absolute improvement in human empathy ratings and the limited conflict-evaluation set, what evidence supports a reliable improvement in human-perceived empathy? Additionally, can the authors provide controlled evaluations using identical transcripts with different vocal emotions to demonstrate that the response changes appropriately because of acoustic cues rather than lexical shortcuts?

3 comments
AuthorsOpenReview ↗

Part I: Clarification of Core Differences from ParaBridge, Cross-Backbone Evidence for Incremental Contribution, Transcript Ablation Justifying Affect Attribution, and the Role of Speech-Token Supervision

We thank the reviewer for the careful and incisive review. We will directly address the detailed issues, while clarifying several key criticisms that do not align with our method, controlled settings, or existing evidence, supported by the corresponding results.

W1&Q3&Q2. (1) We note that our method does resemble ParaBridge’s on-policy privileged-distillation recipe, and we don't claim this training recipe itself as novel. We also address the perception–behavior gap but under different assumption. As shown in Table 1 and Appendix A, our linear probes confirm user affect is already accurately decodable from the audio representation fed to the LM. The core issue is thus not a total lack of affect perception, but difficulty using it to generate responses. ParaBridge assumes this capability pre-exists and only needs elicitation: a generic paralinguistic scaffold induces the target behavior, which is distilled into a scaffold-free student. Its supervision thus transfers only prompt-elicitable behavior and fails when generic prompting cannot evoke the target.

On our tested S2S backbones, generic instructions failed to translate decodable affect into appropriate responses (shown in Figure 1), consistent with ParaBridge’s own second-backbone results showing only marginal MiMo-Audio gains. Our teacher instead explicitly receives perceived input affect, enabling both elicitation and acquisition of affect-conditioned empathy even in models lacking this capability. We verify this via cross-training each backbone (Parabridge uses MiMo-Audio-7B-Instruct and Qwen3-Omni-30B-A3B-Thinking) under the other’s framework:

BackboneTrainingC4C1-C4EmoAlignVESzhenAcousticOverall
MiMo-Audio-7B-Instruct-3.65344.025428.292.678335.6549.5153.446.4
MiMo-Audio-7B-Instruct+ours4.26484.703633.713.457543.9659.6758.852.6
Qwen3-Omni-30B-A3B-Thinking-3.93144.4260------
Qwen3-Omni-30B-A3B-Thinking+ours4.81534.9338------
Fun-Audio-Chat-3.88324.355229.912.971243.5760.5454.443.2
Fun-Audio-Chat+ParaBridge prompt3.87234.354429.452.973443.6259.9653.243.2
Fun-Audio-Chat+ours4.32484.591634.623.817656.2571.7960.254.2
GLM-4-Voice-3.08803.766026.822.564126.9635.5459.446.6
GLM-4-Voice+ParaBridge prompt3.07813.765626.182.565326.8835.7859.246.4
GLM-4-Voice+ours3.63044.106432.023.120028.0445.3662.251.6

As shown, our method transfers to ParaBridge's backbones, whereas ParaBridge's recipe cannot equip affect-blind models with this capability. Moreover, ParaBridge evaluates only text replies, but our work covers both text and speech responses, which matters for natural human–model interaction.

(2) Regarding the transcript, we initially included it purely for completeness—as if providing it would make the teacher infallible—alongside the explicit emotion labels. In testing, however, we found that the transcript makes no difference to our study of empathetic spoken replies, and we admittedly overlooked removing this condition. We thank the reviewer for catching this. Results with and without the transcript:

BackboneTrainingC4C1-C4EmoAlignVESzhenAcousticOverall
Fun-Audio-Chat+ours (full)4.32484.591634.623.817656.2571.7960.254.2
Fun-Audio-Chat+ours, no transcript4.33194.608234.753.825656.5071.9860.254.4
GLM-4-Voice+ours (full)3.63044.106432.023.120028.0445.3662.251.6
GLM-4-Voice+ours, no transcript3.63574.112932.133.127128.4545.7262.051.8

The transcript indeed has no effect, since modern speech LLMs already transcribe user speech content reliably. Moreover:

BackboneTrainingC4C1-C4EmoAlignVESzhenAcousticOverall
Fun-Audio-Chat+ParaBridge prompt3.87234.354429.452.973443.6259.9653.243.2
Fun-Audio-Chat+ours4.32484.591634.623.817656.2571.7960.254.2
GLM-4-Voice+ParaBridge prompt3.07813.765626.182.565326.8835.7859.246.4
GLM-4-Voice+ours3.63044.106432.023.120028.0445.3662.251.6

Training our model with a ParaBridge-style teacher yields no improvement, even though the adapters and student parameters are identical. Thus, these results further confirm that the distributional difference in Eq. (6)–(7) is indeed attributable to the use of affect.

(3) On the independent effect of speech-token supervision: with text-only supervision the student's speech replies do change, and the measured quality differences are not significant. However, without speech-token supervision the student produced 25 invalid speech replies at test time, versus 0 with it. Speech-token supervision is therefore necessary for producing stable, complete speech responses.

AuthorsOpenReview ↗

Part II: Evidence Ruling Out the Lexical-Valence Shortcut, Revised Rigorous Formulation of Equations (6)–(7), and Validation of Affect Attribution

We thank the reviewer for the careful and incisive review. We will directly address the detailed issues, while clarifying several key criticisms that do not align with our method, controlled settings, or existing evidence, supported by the corresponding results.

Q3.

BackboneTrainingC4C1-C4EmoAlignVESzhenAcousticOverall
Fun-Audio-Chat-3.88324.355229.912.971243.5760.5454.443.2
Fun-Audio-Chat+ours, transcript-only3.89254.347229.882.974342.6960.4454.443.4
Fun-Audio-Chat+ours, emotion-only4.33194.608234.753.825656.5071.9860.254.4
Fun-Audio-Chat+ours, combined4.32484.591634.623.817656.2571.7960.254.2
GLM-4-Voice-3.08803.766026.822.564126.9635.5459.446.6
GLM-4-Voice+ours, transcript-only3.07873.792326.922.561027.0635.3859.246.6
GLM-4-Voice+ours, emotion-only3.63574.112932.133.127128.4545.7262.051.8
GLM-4-Voice+ours, combined3.63044.106432.023.120028.0445.3662.251.6

Here emotion-only means the teacher receives audio and perceived affect without a transcript and combined adds the transcript. Transcript has no effect on our study: giving the teacher only the transcript changes nothing over base on either backbone, and removing it from the full privilege costs nothing either—the supervision is carried entirely by perceived affect.

W1&Q2. With ~91.2% of training samples pairing positive words with downcast delivery, the model could in principle learn an inverse lexical-valence heuristic without attending to vocal emotion. That is not the case. Our training data spans many scenarios, and learning to genuinely understand emotional differences is simpler than memorizing such heuristics—as the experiments confirm. Our auxiliary human evaluation (Appendix H) uses 20 recordings from 10 sentence pairs, each sentence read once in each of two opposite emotions. Our method reduces text-dominated responses to 41.7% vs. 70.8% for the base model, showing it relies less on text alone, and raises vocal-affect awareness to 33.3% vs. 25.0%, showing it attends more to vocal emotion. We strongly encourage the reviewer to listen to the paired examples in the supplementary material—they make the voice-conditioned response difference immediately audible.

We further trained a student by SFT on the teacher's output tokens as ground truth (Table 3). The fuller comparison is:

BackboneTrainingC4C1-C4EmoAlignVESzhenAcousticOverall
Fun-Audio-Chat-3.88324.355229.912.971243.5760.5454.443.2
Fun-Audio-Chat+SFT3.90284.375430.033.005643.4960.3754.043.2
Fun-Audio-Chat+SFT+PO4.12804.355635.423.676042.1458.2156.241.0
Fun-Audio-Chat+ours4.32484.591634.623.817656.2571.7960.254.2

Here PO is affect-conditional preference optimization: chosen/rejected share an identical reply and differ only in whether the preceding affect is this utterance's true emotion or another's (in §5.1&Appendix C). These results show rote memorization alone cannot teach emotion understanding and explicit affect information is what lets the model grasp what drives the change—true for both SFT+PO and our method.

Regarding Eq. (6)–(7), our formulation was indeed not rigorous enough. We thank the reviewer and will revise accordingly. In Eq. (6), e^θ(x)\hat e_\theta(x) served as a notational shorthand for the student's inferred audio affect but was never formally defined. We will remove it entirely—Eq. (6) compares the student's and privileged teacher's conditional distributions directly: L(θ)=Ex, y∼πθ(⋅∣x)[1T∑tJSD(πθ(⋅∣x,y<t) ∥ π(⋅∣x,e,c,y<t))].\mathcal L(\theta)=\mathbb E_{x,\,y\sim\pi_\theta(\cdot\mid x)}\Bigl[\tfrac{1}{T}\sum_{t} \mathrm{JSD}\bigl(\pi_\theta(\cdot\mid x,y_{<t})\,\|\,\pi(\cdot\mid x,e,c,y_{<t})\bigr)\Bigr]. For Eq. (7), we linked zero loss to the student's audio representation being "sufficient for affect" via an iff, which is not rigorous and exceeds what the objective alone can establish. We will replace it with a provable identity: L(θ)=0  ⟺  πθ(⋅∣x,y<t)=π(⋅∣x,e,c,y<t) almost surely on prefixes the student actually generates during training.\mathcal L(\theta)=0 \iff \pi_\theta(\cdot\mid x,y_{<t})=\pi(\cdot\mid x,e,c,y_{<t})\ \text{almost surely on prefixes the student actually generates during training}. The argument takes two steps. The loss averages non-negative JSD terms: (⟸) if the distributions are equal almost surely on these prefixes, every JSD term vanishes and thus L(θ)=0\mathcal L(\theta)=0; (⟹) conversely, L(θ)=0\mathcal L(\theta)=0 forces each non-negative term to be zero almost surely, and JSD$(P\Vert Q)=0\iff P=Q$. The meaning is strictly limited: when the loss converges to zero, the student reproduces the privileged teacher's conditional distribution position by position, from audio alone, on the prefixes it actually visits. As for the 2 specific confounds you name—the teacher's transcript and the student's LoRA parameters—both are directly ruled out by the aforementioned experiments. The gains indeed track the affect privilege.

AuthorsOpenReview ↗

Part III: Clarification of the Table 3/17 Labeling Inconsistency, Justification of Evaluation Metrics and Human Evaluation Setup, and Evidence of Acoustic-Cue Driven Response Changes

We thank the reviewer for the careful and incisive review. We will directly address the detailed issues, while clarifying several key criticisms that do not align with our method, controlled settings, or existing evidence, supported by the corresponding results.

W2&Q1&Q4. As for the Table 3/17 inconsistency, we thank the reviewer for catching it: the second block of Table 17 carries an incorrect comparator label—no checkpoint, data, or aggregation difference is involved. The block should read "OPD-NOQAFF vs. OPD with affect readout," not "OPD vs. SFT+PO": its reported differences are 4.3248−4.2480=+0.07684.3248-4.2480=+0.0768 (C4) and 34.6228−33.9688=+0.654034.6228-33.9688=+0.6540 (EmoAlign), whereas the actual OPD vs. SFT+PO differences implied by Table 3 are +0.1968+0.1968 and −0.7953-0.7953, matching your calculation. This is a labeling slip in one block. Every number in Table 3 and every conclusion drawn from Table 17 stands. We will fix the label in the revision.

On the metrics, we respectfully disagree. Empathetic spoken response evaluation is inherently subjective with no universal benchmark, so we use multiple complementary metrics. LLM-judge metrics such as Gemini suffer from text preference, length bias, and prompt sensitivity, so no single metric is reliable. We include human evaluation as an auxiliary third-party reference, triangulating across automatic metrics and human listeners rather than relying on any single measure. Our metrics cover both speech quality and expressiveness (VES, NISQA) and direct alignment with user vocal expression (C4, EmoAlign). The claim that our metrics do not directly observe the user’s voice is thus inaccurate.

Regarding the human evaluation: raters were recruited internally via an internal forum as unpaid volunteers from teams independent of the research group, and rating was fully blind to system identity and sample pairing. Inter-annotator agreement (Krippendorff's α\alpha, observed ratings only): empathy 0.431, conflict judgment 0.381, conflict-conditioned H-Conf 0.398—the latter is gated by the previous question with substantial missingness and is reported only as auxiliary. Paired 95% CIs and significance tests (sentence-cluster bootstrap, exact sign-flip) are reported below:

MetricBaseOPDΔ95% CIp
H-Emp3.85003.8833+0.0333[-0.2333,+0.3333]0.914
H-Conf (imputed)3.37503.4167+0.0417[-0.4167,+0.5000]1.000
Awareness (imputed)25.0%33.3%+8.3pp[-16.7pp,+33.3pp]0.766
Text-dom. (imputed)70.8%41.7%-29.2pp[-58.3pp,0.0pp]0.180
Awareness (observed)30.0%40.0%+10.0pp[-19.0pp,+42.1pp]0.516
Text-dom. (observed)65.0%30.0%-35.0pp[-70.0pp,0.0pp]0.148

While no comparison reaches statistical significance, the human evaluation plays only an auxiliary role. Core conclusions rest on automatic results across three benchmarks, and this study only verifies whether the gains are perceptible to real listeners. All human metrics trend in our favor, consistent with the results of the automatic evaluation. Key behavioral items show clear directional improvements aligned with our core claims. The consistent positive trend across both evaluation modes confirms the effect. The lack of significance stems from the small sample size and the effect direction itself is consistent across all four metrics. We will accordingly scope the human-evaluation claims (abstract, contributions, and conclusion) from "confirm the improvement" to what the statistics support: directional improvements on conflict handling and voice-conditioned responding.

The requested controlled evaluation—identical transcripts under different vocal emotions—is already part of the paper: the human evaluation in Appendix H uses 20 recordings of 10 sentences, each read once with matching and once with opposite emotion on identical transcripts, and the responses do change with vocal emotion rather than with the words. We strongly encourage the reviewer to listen to the four paired examples in the supplementary material—they make the voice-conditioned difference immediately audible.

Reviewer_2ui2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses al issue in speech dialogue models: while a model may be capable of detecting emotion in a user’s voice, it often relies excessively on the textual content when generating responses. The authors propose using OPD to tackle this: during training, a teacher model is provided with both the user’s emotional state and the transcribed text, while a student model, which receives only the raw audio, is trained to mimic the teacher’s output distribution. This enables the student model to learn how to adjust both the textual content and the vocal expression of its responses based on the emotion conveyed in the user’s speech.

Critical strengths

1.The method design is simple and has practical value. Using the same pre-trained model as teacher and student, distillation is achieved through privileged information during training, and no additional emotion recognition module is required during inference.

Critical weaknesses
  1. Synthetic training data may limit generalization capability. The training data consists entirely of TTS-synthesized content. In real-world conversations, emotions tend to be more ambiguous, blended, and dynamic; they do not always manifest as clear-cut, semantic-vocal conflicts.
  2. The scale of human evaluation is too small, resulting in limited actual improvement. The human evaluation involved only 20 audio recordings, with just 8 undergoing in-depth conflict analysis; each response was rated by 3 listeners. As shown in Table 15, the empathy score shifted from 3.85 to 3.88, a very marginal improvement. 3.Missing evaluation of general capability retention. While the proposed OPD method improves empathetic speech-to-speech dialogue performance, the paper does not investigate whether these gains come at the expense of the backbone's general capabilities.
Questions for authors

see above

1 comment
AuthorsOpenReview ↗

Justification of Synthetic Training Data and Generalization, Clarification of Human Evaluation Scope, and New Results on General Capability Retention

We thank the reviewer for the careful review.

W1. Regarding training data, we respectfully disagree that synthetic content inherently limits generalization. This claim lacks empirical support and conflicts with our results. We use synthetic speech to fix transcripts and speakers while precisely controlling vocal affect, disentangling lexical content from vocal emotion to learn voice-conditioned responses rather than memorize conflict templates. Consistent improvements across multiple public benchmarks (not limited to conflict detection) show the learned behavior transfers beyond the training distribution. Paired examples in the supplementary material further demonstrate our method responds appropriately to both synthetic and natural human speech.

Controlled synthesis is popular in spoken LLM research because high-quality real paired speech with fine-grained vocal-affect annotations is scarce and difficult to scale across speakers, emotion categories, intensities, and speaking styles. GLM-4-Voice (Zeng et al., 2024) and Fun-Audio-Chat (Team, Tongyi Fun, et al., 2025) synthesize large-scale text data for pre-training or full-parameter SFT. Closely related to our task, OpenS2S (Wang et al., 2025) combines real emotional seeds, voice cloning, and instruction-controlled TTS; Empathy Omni (Wang et al., 2025) constructs emotional-dialogue data with controllable TTS; CogAudio-LLM (Zhao et al., 2026) controls emotion category and intensity under controlled text and ALPO (Lin et al., 2026) performs preference optimization on 140K TTS-based training samples... Thus, we believe synthetic fine-tuning data alone doesn't necessarily limit generalization.

W2. (1) Our human evaluation is indeed modest in scale because it was designed as auxiliary validation. Using identical transcripts and speakers while varying only vocal affect, it includes 20 user inputs, 120 system responses, and 360 independent ratings. It tests whether real listeners show behavioral changes consistent with the automatic evaluation and the core conclusions rest on automatic results across 3 public benchmarks. The 8 cases used for conflict analysis were not post hoc favorable selections: 8 of the 10 predefined conflict cases were confirmed by at least 2 of the 3 raters and were the only cases containing an evaluable semantic–vocal conflict. This subset covers 8×6=48 responses and 144 ratings.

On the behavioral measures closest to our claims, our method reduces text-dominated responses from 70.8% to 41.7%, indicating less reliance on text alone, and increases vocal-affect awareness from 25.0% to 33.3%, indicating greater attention to vocal emotion. All human-evaluation metrics trend in our favor. The paired examples in the supplementary material also directly demonstrate how the response changes with vocal emotion for the same transcript, and we strongly encourage the reviewer to listen to them. Accordingly, we will scope the human-evaluation claims in the abstract, contributions, and conclusion from "confirm the improvement" to what the statistics support: directional improvements in conflict handling and voice-conditioned responding.

(2) Following your advice, we evaluate both S2S backbones used in our study on MMSU and MMAU-Pro under the Fun-Audio-Chat (Team, Tongyi Fun, et al., 2025) evaluation setting. The results show that the empathy gains of our method do not come at the expense of the backbone’s general audio-understanding and reasoning capabilities.

BackboneTrainingMMSUMMAU-Pro
Fun-Audio-Chat-68.158.2
Fun-Audio-Chat+ours68.058.2
GLM-4-Voice-34.940.6
GLM-4-Voice+ours35.140.7
Reviewer_xo2YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies why end-to-end S2S dialogue models answer the words of an utterance but largely ignore how it is spoken. The authors support this observation through linear probing and comparisons between audio-only and affect-informed prompting.

To address this gap, they propose an OPD framework using a frozen privileged teacher that receives the user's transcript and perceived affect, while the student receives only raw audio. The teacher provides dense distributional supervision through Jensen–Shannon divergence over both text and speech tokens along student-generated trajectories. Experiments on Fun-Audio-Chat and GLM-4-Voice demonstrate consistent improvements across EchoMind, SpeechParaling-Bench, and VocalBench-zh, with additional human evaluation supporting the gains. The paper argues that OPD improves empathetic response generation without requiring additional inference-time modules.

Critical strengths
  • The conflict between user semantics and nonsemantic information during interaction is a concern in the field of speech models.
  • The proposed method demonstrate improvement over baseline on the benchmarks
Critical weaknesses
  • Missing controls limit attribution of the empirical gains.

    • Appendix C shows that the student receives an affect-oriented system instruction, but Table 2 does not clarify whether the backbone baselines use the same prompt. Also, results for the privileged teacher and a simple ASR+SER-conditioned backbone would also clarify the distillation gap and the advantage over a deployable alternative.
  • The claimed theoretical advantage of OPD is not established.

    • Contribution 3 says "we prove" that the OPD loss is zero only if the audio representation encodes affect. Yet Eq. 7 is neither formally defined nor proved. Eq. 6’s cancellation argument is also unsupported: the student has active LoRA adapters, and the teacher receives both affect and transcript, so their divergence need not isolate emotion. Moreover, reply-level SFT can require acoustic information when identical transcripts have different affect-dependent targets; this property is not unique to OPD.
    • The experiments show a trade-off: SFT+PO-t leads on all four EchoMind metrics, although it produces 12 invalid outputs and trails OPD on the other benchmarks (Table 3). Although §5.1 mentions teacher-target SFT, standalone quantitative results are absent, while the reported SFT+PO variants additionally use an affect readout and affect-condition preference loss. Matched teacher-target SFT and off-policy distributional distillation are needed to isolate the on-policy contribution.
Questions for authors
  • Were the Table 2 backbone and OPD models evaluated with identical system prompts? If not, are prompt-matched results available?
  • How is affect sufficiency formally defined in Eq. 7, and what assumptions justify Eqs. 6–7 given student-only LoRA, the teacher’s additional transcript, and possible lexical shortcuts?
  • What are the standalone results for teacher-target SFT mentioned in §5.1? Are matched off-policy distillation results available to distinguish the benefit of on-policy sampling from distributional supervision?
8 comments
AuthorsOpenReview ↗

Part I: Prompt-Matched Baseline Results, Transcript Ablation, and New Teacher & SER-Conditioned Scores Clarifying Gain Attribution

We thank the reviewer for the careful review.

W1&Q1. We confirm that the backbone baselines reported in the paper indeed didn't use the identical system instruction, because we had found that it barely affects performance. We admittedly overlooked the strict control of this variable, and we thank the reviewer for pointing it out. In fact, we can provide the prompt-matched results:

BackboneTrainingC4C1-C4EmoAlignVESzhenAcousticOverall
Fun-Audio-Chat-3.88324.355229.912.971243.5760.5454.443.2
Fun-Audio-Chat- (with same system grounding)3.89124.358630.042.967943.0861.0654.443.4
GLM-4-Voice-3.08803.766026.822.564126.9635.5459.446.4
GLM-4-Voice- (with same system grounding)3.09053.774326.542.562227.1835.2659.246.2

The results show that this single grounding instruction has almost no effect on the backbone's performance and the real gap indeed comes from training.

In addition, we need to clarify the transcript. We initially included it purely for completeness—as if providing it would make the teacher infallible—alongside the explicit emotion labels. In testing, however, we found that the transcript makes no difference to our study of empathetic spoken replies, and we admittedly overlooked removing this condition. We thank the reviewer for catching this. Results with and without the transcript:

BackboneTrainingC4C1-C4EmoAlignVESzhenAcousticOverall
Fun-Audio-Chat+ours (full)4.32484.591634.623.817656.2571.7960.254.2
Fun-Audio-Chat+ours, no transcript4.33194.608234.753.825656.5071.9860.254.4
GLM-4-Voice+ours (full)3.63044.106432.023.120028.0445.3662.251.6
GLM-4-Voice+ours, no transcript3.63574.112932.133.127128.4545.7262.051.8

The transcript indeed has no effect, since modern speech LLMs already transcribe user speech content reliably.

Moreover, following your advice, we now report the teacher's inference scores. Since the transcript is effectively inconsequential, we additionally test only the SER-conditioned variant, i.e., replacing the teacher's affect description with the emotion label predicted by emotion2vec:

BackboneTrainingC4C1-C4EmoAlignVESzhenAcousticOverall
Fun-Audio-Chat- (teacher inference)4.53764.740835.894.016258.3673.6462.257.2
Fun-Audio-Chat- (with SER label)4.11544.392731.843.463549.7866.5156.047.4
GLM-4-Voice- (teacher inference)3.86734.341834.213.317330.2447.5964.252.6
GLM-4-Voice- (with SER label)3.39683.927029.742.837527.6339.9860.648.8

These results demonstrate the effectiveness of our constructed training data and of our method.

AuthorsOpenReview ↗

Part II: Formal Revision of Equations (6)–(7) with Validated Affect Attribution, Standalone Teacher-Target SFT Results, and Off-Policy Distillation Comparisons Isolating On-Policy Contribution

W2&Q2. As discussed above, whether the transcript is provided has no effect on our study. In addition, we compare a model trained with a ParaBridge-style prompt against our method:

BackboneTrainingC4C1-C4EmoAlignVESzhenAcousticOverall
Fun-Audio-Chat+ParaBridge prompt3.87234.354429.452.973443.6259.9653.243.2
Fun-Audio-Chat+ours4.32484.591634.623.817656.2571.7960.254.2
GLM-4-Voice+ParaBridge prompt3.07813.765626.182.565326.8835.7859.246.4
GLM-4-Voice+ours3.63044.106432.023.120028.0445.3662.251.6

Training our model with a ParaBridge-style teacher yields no improvement, even though the adapters and student parameters are identical. Thus, these results further confirm that the distributional difference in Eq. (6)–(7) is indeed attributable to the use of affect.

Regarding Eq. (6)–(7), our formulation was indeed not rigorous enough. We thank the reviewer and will revise accordingly. In Eq. (6), e^θ(x)\hat e_\theta(x) served as a notational shorthand for the student's inferred audio affect but was never formally defined. We will remove it entirely—Eq. (6) compares the student's and privileged teacher's conditional distributions directly: L(θ)=Ex, y∼πθ(⋅∣x)[1T∑tJSD(πθ(⋅∣x,y<t) ∥ π(⋅∣x,e,c,y<t))].\mathcal L(\theta)=\mathbb E_{x,\,y\sim\pi_\theta(\cdot\mid x)}\Bigl[\tfrac{1}{T}\sum_{t} \mathrm{JSD}\bigl(\pi_\theta(\cdot\mid x,y_{<t})\,\|\,\pi(\cdot\mid x,e,c,y_{<t})\bigr)\Bigr]. For Eq. (7), we linked zero loss to the student's audio representation being "sufficient for affect" via an iff, which is not rigorous and exceeds what the objective alone can establish. We will replace it with a provable identity: L(θ)=0  ⟺  πθ(⋅∣x,y<t)=π(⋅∣x,e,c,y<t) almost surely on prefixes the student actually generates during training.\mathcal L(\theta)=0 \iff \pi_\theta(\cdot\mid x,y_{<t})=\pi(\cdot\mid x,e,c,y_{<t})\ \text{almost surely on prefixes the student actually generates during training}. The argument takes two steps. The loss averages non-negative JSD terms: (⟸) if the distributions are equal almost surely on these prefixes, every JSD term vanishes and thus L(θ)=0\mathcal L(\theta)=0; (⟹) conversely, L(θ)=0\mathcal L(\theta)=0 forces each non-negative term to be zero almost surely, and JSD$(P\Vert Q)=0\iff P=Q$. The meaning is strictly limited: when the loss converges to zero, the student reproduces the privileged teacher's conditional distribution position by position, from audio alone, on the prefixes it actually visits. As for the 2 specific confounds you name—the teacher's transcript and the student's LoRA parameters—both are directly ruled out by the aforementioned experiments. The gains indeed track the affect privilege.

We further trained a student by SFT on the teacher's output tokens as ground truth (Table 3). The fuller comparison is:

BackboneTrainingC4C1-C4EmoAlignVESzhenAcousticOverall
Fun-Audio-Chat-3.88324.355229.912.971243.5760.5454.443.2
Fun-Audio-Chat+SFT3.90284.375430.033.005643.4960.3754.043.2
Fun-Audio-Chat+SFT+PO4.12804.355635.423.676042.1458.2156.241.0
Fun-Audio-Chat+ours4.32484.591634.623.817656.2571.7960.254.2

Here PO is affect-conditional preference optimization: chosen/rejected share an identical reply and differ only in whether the preceding affect is this utterance's true emotion or another's (in §5.1&Appendix C). These results show rote memorization alone cannot teach emotion understanding and explicit affect information is what lets the model grasp what drives the change. While this property is not unique to OPD, in our experiments response-level SFT requires preference structure formed with other samples' affect to force the model to acquire such capability, whereas OPD transfers affect dependence directly through the teacher's dense per-position distributional supervision, with no pairing construction needed.

W2&Q3. In our view stable responses do matter, so our work vs SFT+PO-t isn't a simple trade-off. The standalone results for teacher-target SFT are already shown above, demonstrating that SFT alone cannot teach the model to exploit affect in its responses. Following your advice, we provide matched off-policy distillation results:

BackboneTrainingC4C1-C4EmoAlignVESzhenAcousticOverall
Fun-Audio-Chat-3.88324.355229.912.971243.5760.5454.443.2
Fun-Audio-Chat+ours (off-policy)4.10584.384630.923.427447.2364.0955.245.8
Fun-Audio-Chat+ours (on-policy)4.32484.591634.623.817656.2571.7960.254.2
GLM-4-Voice-3.08803.766026.822.564126.9635.5459.446.6
GLM-4-Voice+ours (off-policy)3.28763.846127.952.703927.5036.9760.047.2
GLM-4-Voice+ours (on-policy)3.63044.106432.023.120028.0445.3662.251.6

Both distillation methods are effective, while OPD contributes more, indicating that behavior should be shaped on the student's own state distribution.

Thank you for the clarifications and additional results. Two concerns remain:

  1. I know that synthetic data is useful. Appendix B states that 91.2% of training samples contain lexical–vocal conflicts. Isn't this too far from the real distribution?

  2. Section 3.3 discusses RL only with teacher log-likelihood as the reward. Although the text repeatedly explains why no RL arm was included, using RL to train empathy capability is a more mainstream approach, and I believe direct comparison is necessary.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.