- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes an on-policy distillation (OPD) framework for improving empathetic speech-to-speech (S2S) dialogue. Motivated by the perception–behavior gap, where speech models may encode paralinguistic information without appropriately incorporating it into their responses, the authors use a frozen privileged teacher that receives the user's speech, transcript, and explicit emotion description. A student model receives only the speech input and learns from the teacher's token-level probability distributions over both textual and speech outputs through Jensen–Shannon divergence.
The method is trained using 34,305 synthetic Chinese and English utterances and evaluated on two S2S backbones across three benchmarks. The authors report improvements over the original backbones on several measures of response appropriateness and vocal expressiveness, alongside a small-scale human evaluation. They further provide linear probing experiments and theoretical arguments intended to explain why the proposed objective encourages acoustic emotion grounding.
Critical strengths
- The proposed training procedure improves the reported performance of two different S2S backbones across several automatic evaluation metrics, including vocal expressiveness and speech-response appropriateness. Incorporating speech-token-level supervision is also potentially useful for extending privileged distillation to end-to-end spoken response generation.
Critical weaknesses
-
The proposed framework closely resembles ParaBridge, which already addresses the perception–behavior gap through privileged on-policy self-distillation, with student-generated trajectories and dense token-level distributional supervision. The main distinctions here appear to be explicitly providing emotion labels and transcripts to the teacher, freezing the teacher, and extending the objective to speech tokens. However, the paper lacks a controlled comparison against a ParaBridge-style approach using the same backbone, training data, and computational budget. Consequently, the incremental contribution is not convincingly established. More importantly, I find the central theoretical justification problematic. Equations (6)–(7) claim that the distillation objective isolates the contribution of emotion and reaches zero only when the student's audio-derived representation is sufficient for the user's affect. However, the teacher receives both the transcript and emotion description, while the student receives only audio. Furthermore, the teacher and student have different effective parameters because the student contains trainable adapters. Therefore, their distributional differences cannot be attributed exclusively to emotion. The objective may also be minimized through alternative shortcuts rather than genuine acoustic emotion grounding. This concern is particularly relevant because approximately 91.2% of the synthetic training examples deliberately pair positive lexical content with downcast delivery or negative lexical content with cheerful delivery. Such a systematic relationship could allow a model to learn an inverse lexical-valence heuristic without reliably attending to vocal emotion. The paper does not sufficiently rule out this possibility.
-
I identified an important numerical inconsistency between Table 3 and Table 17. Table 3 reports an EchoMind C4 score of 4.1280 for SFT+PO and 4.3248 for OPD, implying a difference of +0.1968. However, Table 17 reports a paired mean difference of only +0.0768 for OPD versus SFT+PO. Similarly, Table 3 reports EmoAlign scores of 35.42 and 34.62, implying that OPD performs worse by 0.80, whereas Table 17 reports a positive difference of +0.6540 in favor of OPD. If these tables refer to the same checkpoints and evaluation items, the reported differences are mathematically inconsistent. The paper does not clearly explain any difference in the evaluated data, checkpoints, or aggregation procedures that could reconcile these numbers. This substantially undermines confidence in the reported statistical comparisons, particularly because Table 17 is intended to substantiate the effectiveness of OPD relative to alternative training objectives. Furthermore, the evaluation does not sufficiently establish that the improvements reflect genuine acoustic emotion understanding. Several automatic metrics assess the generated response or its emotional expression without directly observing the user's original vocal delivery. The human evaluation is also very limited, involving only 20 utterances and eight retained conflict examples. The empathy rating increases only from 3.85 to 3.88, without reported inter-annotator agreement, confidence intervals, or significance tests for the human comparisons.
Questions for authors
-
Could the authors explain the discrepancies between Tables 3 and 17? Specifically, why is the OPD-versus-SFT+PO EchoMind C4 difference reported as +0.0768 in Table 17 when the corresponding Table 3 scores imply +0.1968? Likewise, why does the EmoAlign comparison change direction between the two tables? Were different checkpoints, evaluation subsets, or aggregation procedures used? Please identify the exact source of these differences and provide corrected or reconciled results.
-
Could the authors provide a rigorous justification for Equation (7), explicitly accounting for the additional transcript available to the teacher and the differing effective teacher–student parameters? Under what assumptions does minimizing the proposed divergence necessarily require learning acoustic affect, rather than reproducing teacher behavior through lexical information or other shortcuts? In particular, how can this claim be reconciled with the strong inverse relationship between lexical and vocal valence in the training data?
-
What is the specific methodological contribution beyond ParaBridge, and what experimental evidence isolates its effect? A controlled comparison using the same backbone, dataset, and training budget would materially affect my assessment. In addition, can the authors provide downstream results isolating emotion-only, transcript-only, and combined teacher privileges, as well as the independent contribution of speech-token supervision?
-
Could the authors clarify how the human evaluators were recruited and report inter-annotator agreement, confidence intervals, and statistical comparisons between OPD and its original backbone? Given the small absolute improvement in human empathy ratings and the limited conflict-evaluation set, what evidence supports a reliable improvement in human-perceived empathy? Additionally, can the authors provide controlled evaluations using identical transcripts with different vocal emotions to demonstrate that the response changes appropriately because of acoustic cues rather than lexical shortcuts?
3 comments
Part II: Evidence Ruling Out the Lexical-Valence Shortcut, Revised Rigorous Formulation of Equations (6)–(7), and Validation of Affect Attribution
We thank the reviewer for the careful and incisive review. We will directly address the detailed issues, while clarifying several key criticisms that do not align with our method, controlled settings, or existing evidence, supported by the corresponding results.
Q3.
| Backbone | Training | C4 | C1-C4 | EmoAlign | VES | zh | en | Acoustic | Overall |
|---|---|---|---|---|---|---|---|---|---|
| Fun-Audio-Chat | - | 3.8832 | 4.3552 | 29.91 | 2.9712 | 43.57 | 60.54 | 54.4 | 43.2 |
| Fun-Audio-Chat | +ours, transcript-only | 3.8925 | 4.3472 | 29.88 | 2.9743 | 42.69 | 60.44 | 54.4 | 43.4 |
| Fun-Audio-Chat | +ours, emotion-only | 4.3319 | 4.6082 | 34.75 | 3.8256 | 56.50 | 71.98 | 60.2 | 54.4 |
| Fun-Audio-Chat | +ours, combined | 4.3248 | 4.5916 | 34.62 | 3.8176 | 56.25 | 71.79 | 60.2 | 54.2 |
| GLM-4-Voice | - | 3.0880 | 3.7660 | 26.82 | 2.5641 | 26.96 | 35.54 | 59.4 | 46.6 |
| GLM-4-Voice | +ours, transcript-only | 3.0787 | 3.7923 | 26.92 | 2.5610 | 27.06 | 35.38 | 59.2 | 46.6 |
| GLM-4-Voice | +ours, emotion-only | 3.6357 | 4.1129 | 32.13 | 3.1271 | 28.45 | 45.72 | 62.0 | 51.8 |
| GLM-4-Voice | +ours, combined | 3.6304 | 4.1064 | 32.02 | 3.1200 | 28.04 | 45.36 | 62.2 | 51.6 |
Here emotion-only means the teacher receives audio and perceived affect without a transcript and combined adds the transcript. Transcript has no effect on our study: giving the teacher only the transcript changes nothing over base on either backbone, and removing it from the full privilege costs nothing either—the supervision is carried entirely by perceived affect.
W1&Q2. With ~91.2% of training samples pairing positive words with downcast delivery, the model could in principle learn an inverse lexical-valence heuristic without attending to vocal emotion. That is not the case. Our training data spans many scenarios, and learning to genuinely understand emotional differences is simpler than memorizing such heuristics—as the experiments confirm. Our auxiliary human evaluation (Appendix H) uses 20 recordings from 10 sentence pairs, each sentence read once in each of two opposite emotions. Our method reduces text-dominated responses to 41.7% vs. 70.8% for the base model, showing it relies less on text alone, and raises vocal-affect awareness to 33.3% vs. 25.0%, showing it attends more to vocal emotion. We strongly encourage the reviewer to listen to the paired examples in the supplementary material—they make the voice-conditioned response difference immediately audible.
We further trained a student by SFT on the teacher's output tokens as ground truth (Table 3). The fuller comparison is:
| Backbone | Training | C4 | C1-C4 | EmoAlign | VES | zh | en | Acoustic | Overall |
|---|---|---|---|---|---|---|---|---|---|
| Fun-Audio-Chat | - | 3.8832 | 4.3552 | 29.91 | 2.9712 | 43.57 | 60.54 | 54.4 | 43.2 |
| Fun-Audio-Chat | +SFT | 3.9028 | 4.3754 | 30.03 | 3.0056 | 43.49 | 60.37 | 54.0 | 43.2 |
| Fun-Audio-Chat | +SFT+PO | 4.1280 | 4.3556 | 35.42 | 3.6760 | 42.14 | 58.21 | 56.2 | 41.0 |
| Fun-Audio-Chat | +ours | 4.3248 | 4.5916 | 34.62 | 3.8176 | 56.25 | 71.79 | 60.2 | 54.2 |
Here PO is affect-conditional preference optimization: chosen/rejected share an identical reply and differ only in whether the preceding affect is this utterance's true emotion or another's (in §5.1&Appendix C). These results show rote memorization alone cannot teach emotion understanding and explicit affect information is what lets the model grasp what drives the change—true for both SFT+PO and our method.
Regarding Eq. (6)–(7), our formulation was indeed not rigorous enough. We thank the reviewer and will revise accordingly. In Eq. (6), served as a notational shorthand for the student's inferred audio affect but was never formally defined. We will remove it entirely—Eq. (6) compares the student's and privileged teacher's conditional distributions directly: For Eq. (7), we linked zero loss to the student's audio representation being "sufficient for affect" via an iff, which is not rigorous and exceeds what the objective alone can establish. We will replace it with a provable identity: The argument takes two steps. The loss averages non-negative JSD terms: (⟸) if the distributions are equal almost surely on these prefixes, every JSD term vanishes and thus ; (⟹) conversely, forces each non-negative term to be zero almost surely, and JSD$(P\Vert Q)=0\iff P=Q$. The meaning is strictly limited: when the loss converges to zero, the student reproduces the privileged teacher's conditional distribution position by position, from audio alone, on the prefixes it actually visits. As for the 2 specific confounds you name—the teacher's transcript and the student's LoRA parameters—both are directly ruled out by the aforementioned experiments. The gains indeed track the affect privilege.
Part III: Clarification of the Table 3/17 Labeling Inconsistency, Justification of Evaluation Metrics and Human Evaluation Setup, and Evidence of Acoustic-Cue Driven Response Changes
We thank the reviewer for the careful and incisive review. We will directly address the detailed issues, while clarifying several key criticisms that do not align with our method, controlled settings, or existing evidence, supported by the corresponding results.
W2&Q1&Q4. As for the Table 3/17 inconsistency, we thank the reviewer for catching it: the second block of Table 17 carries an incorrect comparator label—no checkpoint, data, or aggregation difference is involved. The block should read "OPD-NOQAFF vs. OPD with affect readout," not "OPD vs. SFT+PO": its reported differences are (C4) and (EmoAlign), whereas the actual OPD vs. SFT+PO differences implied by Table 3 are and , matching your calculation. This is a labeling slip in one block. Every number in Table 3 and every conclusion drawn from Table 17 stands. We will fix the label in the revision.
On the metrics, we respectfully disagree. Empathetic spoken response evaluation is inherently subjective with no universal benchmark, so we use multiple complementary metrics. LLM-judge metrics such as Gemini suffer from text preference, length bias, and prompt sensitivity, so no single metric is reliable. We include human evaluation as an auxiliary third-party reference, triangulating across automatic metrics and human listeners rather than relying on any single measure. Our metrics cover both speech quality and expressiveness (VES, NISQA) and direct alignment with user vocal expression (C4, EmoAlign). The claim that our metrics do not directly observe the user’s voice is thus inaccurate.
Regarding the human evaluation: raters were recruited internally via an internal forum as unpaid volunteers from teams independent of the research group, and rating was fully blind to system identity and sample pairing. Inter-annotator agreement (Krippendorff's , observed ratings only): empathy 0.431, conflict judgment 0.381, conflict-conditioned H-Conf 0.398—the latter is gated by the previous question with substantial missingness and is reported only as auxiliary. Paired 95% CIs and significance tests (sentence-cluster bootstrap, exact sign-flip) are reported below:
| Metric | Base | OPD | Δ | 95% CI | p |
|---|---|---|---|---|---|
| H-Emp | 3.8500 | 3.8833 | +0.0333 | [-0.2333,+0.3333] | 0.914 |
| H-Conf (imputed) | 3.3750 | 3.4167 | +0.0417 | [-0.4167,+0.5000] | 1.000 |
| Awareness (imputed) | 25.0% | 33.3% | +8.3pp | [-16.7pp,+33.3pp] | 0.766 |
| Text-dom. (imputed) | 70.8% | 41.7% | -29.2pp | [-58.3pp,0.0pp] | 0.180 |
| Awareness (observed) | 30.0% | 40.0% | +10.0pp | [-19.0pp,+42.1pp] | 0.516 |
| Text-dom. (observed) | 65.0% | 30.0% | -35.0pp | [-70.0pp,0.0pp] | 0.148 |
While no comparison reaches statistical significance, the human evaluation plays only an auxiliary role. Core conclusions rest on automatic results across three benchmarks, and this study only verifies whether the gains are perceptible to real listeners. All human metrics trend in our favor, consistent with the results of the automatic evaluation. Key behavioral items show clear directional improvements aligned with our core claims. The consistent positive trend across both evaluation modes confirms the effect. The lack of significance stems from the small sample size and the effect direction itself is consistent across all four metrics. We will accordingly scope the human-evaluation claims (abstract, contributions, and conclusion) from "confirm the improvement" to what the statistics support: directional improvements on conflict handling and voice-conditioned responding.
The requested controlled evaluation—identical transcripts under different vocal emotions—is already part of the paper: the human evaluation in Appendix H uses 20 recordings of 10 sentences, each read once with matching and once with opposite emotion on identical transcripts, and the responses do change with vocal emotion rather than with the words. We strongly encourage the reviewer to listen to the four paired examples in the supplementary material—they make the voice-conditioned difference immediately audible.
Part I: Clarification of Core Differences from ParaBridge, Cross-Backbone Evidence for Incremental Contribution, Transcript Ablation Justifying Affect Attribution, and the Role of Speech-Token Supervision
We thank the reviewer for the careful and incisive review. We will directly address the detailed issues, while clarifying several key criticisms that do not align with our method, controlled settings, or existing evidence, supported by the corresponding results.
W1&Q3&Q2. (1) We note that our method does resemble ParaBridge’s on-policy privileged-distillation recipe, and we don't claim this training recipe itself as novel. We also address the perception–behavior gap but under different assumption. As shown in Table 1 and Appendix A, our linear probes confirm user affect is already accurately decodable from the audio representation fed to the LM. The core issue is thus not a total lack of affect perception, but difficulty using it to generate responses. ParaBridge assumes this capability pre-exists and only needs elicitation: a generic paralinguistic scaffold induces the target behavior, which is distilled into a scaffold-free student. Its supervision thus transfers only prompt-elicitable behavior and fails when generic prompting cannot evoke the target.
On our tested S2S backbones, generic instructions failed to translate decodable affect into appropriate responses (shown in Figure 1), consistent with ParaBridge’s own second-backbone results showing only marginal MiMo-Audio gains. Our teacher instead explicitly receives perceived input affect, enabling both elicitation and acquisition of affect-conditioned empathy even in models lacking this capability. We verify this via cross-training each backbone (Parabridge uses MiMo-Audio-7B-Instruct and Qwen3-Omni-30B-A3B-Thinking) under the other’s framework:
As shown, our method transfers to ParaBridge's backbones, whereas ParaBridge's recipe cannot equip affect-blind models with this capability. Moreover, ParaBridge evaluates only text replies, but our work covers both text and speech responses, which matters for natural human–model interaction.
(2) Regarding the transcript, we initially included it purely for completeness—as if providing it would make the teacher infallible—alongside the explicit emotion labels. In testing, however, we found that the transcript makes no difference to our study of empathetic spoken replies, and we admittedly overlooked removing this condition. We thank the reviewer for catching this. Results with and without the transcript:
The transcript indeed has no effect, since modern speech LLMs already transcribe user speech content reliably. Moreover:
Training our model with a ParaBridge-style teacher yields no improvement, even though the adapters and student parameters are identical. Thus, these results further confirm that the distributional difference in Eq. (6)–(7) is indeed attributable to the use of affect.
(3) On the independent effect of speech-token supervision: with text-only supervision the student's speech replies do change, and the measured quality differences are not significant. However, without speech-token supervision the student produced 25 invalid speech replies at test time, versus 0 with it. Speech-token supervision is therefore necessary for producing stable, complete speech responses.