acceptodds
Withdrawn from ICLR 2027

DynaEmpathy: Benchmarking and Training Speech Language Models for Interactive Empathetic Spoken Dialogue

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Empathy is a core capability for speech language models, enabling them to respond appropriately to both users' words and vocal signals of affect, urgency, uncertainty, and interpersonal need. Despite progress in spoken-empathy evaluation and alignment, dynamic empathic strategies that adapt to evolving semantic and vocal evidence and user feedback remain insufficiently characterized. To evaluate this dynamic multi-turn empathy in speech language models, we introduce DynaEmpathy, an interactive benchmark grounded in two real-world interaction settings: implicit-affect problem solving and emotion disclosure. Its pipeline combines scenario-grounded seed construction, speech-aware response-contingent user simulation, multi-stage quality control, and trajectory-level evaluation. Conditioned on a private user state and the realized history, the simulator makes assistant decisions consequential for subsequent user behavior and the evaluation context. Our evaluation shows that locally helpful or affectively plausible replies often fail to form a coherent, need-sensitive strategy across turns. To address this gap, we train state-conditioned policies from static turn-level states distilled from simulated interactions. Our responsibility-routed, modality-decomposed GRPO assigns stage-specific semantic rewards to text actions and fidelity and delivery rewards to speech actions, without online user-simulator rollouts. Experiments on the in-domain DynaEmpathy benchmark and out-of-domain spoken-empathy benchmarks show that the method improves the model's empathy in both response content and vocal delivery while retaining aggregate speech-to-text performance on general spoken tasks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Speech Language Models, Empathetic Spoken Dialogue, User Simulator

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_o7QsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DynaEmpathy is a 600-condition benchmark where a simulated user with hidden needs reacts to each spoken reply, in implicit-affect problem solving (PS) and emotion disclosure (ED). The paper also trains Fun-Audio-Chat-8B with SFT and static-state GRPO, which raises its pass rate from 15.0% to 73.7%.

Critical strengths
  1. The benchmark is a useful and carefully checked test of spoken empathy in dialogue. Judge scores track mean human ratings (Spearman 0.784–0.807), and it shows that current models struggle in PS (1.0–27.3% pass).

  2. The training recipe raises the pass rate from 15.0% to 73.7% without online simulator rollouts. It also improves VStyle (4.63 to 4.81) and all five EchoMind metrics while keeping overall speech-to-text scores steady.

Critical weaknesses
  1. The multi-turn gains are shown only on the paper's own benchmark, which shares its pipeline with the training data. Topics are not held out (App. E.2), the PS-O reward closely matches the judge's opening rule (App. C.4, G.1), and the out-of-domain tests are single-turn. This is the main reason for my rating, since the 73.7% pass rate may partly reflect fit to this setup.

  2. The "dynamic" part is not separated from handling emotion in the first turn. In PS, a missed opening emotion caps adaptive empathy at 3 (App. C.4), and PS dialogues average only 2.73–3.85 assistant turns (Table 13). The handled/missed labels are not reported, so it is not clear whether low scores and gains come from later turns.

  3. The benchmark seems to be a small step beyond earlier interactive evaluation. Table 4 shows that MULTI-Bench [1] already runs interactive spoken dialogues with private user profiles. The main additions, such as a simulator that hears the assistant's audio and runtime checks, are not shown to change any score or ranking.

  4. The parts that make the method new are not tested. As Sec. 5.2 notes, the gains support the whole procedure rather than each reward component, and only one base model is used. Without unrouted or joint-reward GRPO baselines on the same data, the value of routing and modality splitting is not clear.

  5. The results rest on one synthetic setup, and the paper notes that its audits do not show fidelity to real users. All user speech is TTS, and the human simulator audit has low agreement on several criteria (α = 0.022–0.182). The simulator also comes from the same Gemini family as the SFT teacher and the topline, so it may favor their style.


References:

[1] Yayue Deng et al. MULTI-Bench: A multi-turn interactive benchmark for assessing emotional intelligence ability of spoken dialogue models. arXiv preprint arXiv:2511.00850, 2025.

[2] Shuiyuan Wang et al. HumDial-EIBench: A human-recorded multi-turn emotional intelligence benchmark for audio language models. arXiv preprint arXiv:2604.11594, 2026.

Questions for authors

Questions:

  1. What are the human scores and pass rates of Fun-Audio-Chat-8B and GRPO on the 60 seeds in your human study? A similar human-rated gain would make the training claim much stronger.
  2. How often does the judge mark each model's PS opening as missed, and what is the PS pass rate when it is handled? Showing that later turns, not the first reply, drive the outcomes would ease my concern about the "dynamic" part.
  3. Does the full method beat unrouted-rubric and joint-reward GRPO with matched data, and does it help on a multi-turn benchmark such as MULTI-Bench [1] or HumDial-EIBench [2]? Gains under these checks would raise my view of the method.
  4. Do scores or rankings change if the simulator gets only the assistant's transcript? Since hearing the assistant's audio is a key difference from MULTI-Bench [1], an effect here would raise my view of the benchmark's novelty.

Suggestions:

  1. Since GRPO and the topline reach about 97% in ED, harder ED cases would keep the benchmark useful.
  2. Please state what will be released, since the simulator and judge are closed models that may change.
Reviewer_pZ86OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces DynaEmpathy, an interactive benchmark and training framework for evaluating dynamic empathy in speech language models (Speech-LLMs). The authors argue that existing single-turn empathy evaluations cannot adequately capture whether models adapt their conversational strategies and vocal delivery to evolving user needs and feedback. Experiments compare several open- and closed-source Speech-LLMs. For Fun-Audio-Chat-8B, the proposed training pipeline increases the benchmark pass rate from 15.0% to 41.3% after SFT and 73.7% after GRPO. The authors additionally report improvements on VStyle and EchoMind, human–automatic evaluation agreement, and simulator-quality audits.

Critical strengths
  1. The paper addresses a meaningful limitation of existing spoken-dialogue evaluations: an appropriate individual response does not necessarily imply effective support throughout an evolving conversation. Integrating response-contingent interaction, private user needs, semantic response assessment, and vocal-delivery evaluation is practically valuable. The comparison across multiple Speech-LLMs provides useful evidence that strong performance on single-turn empathy benchmarks does not necessarily translate into high success rates under the proposed interactive evaluation.

  2. The reported improvement from 15.0% to 73.7% on DynaEmpathy is substantial, with additional gains on VStyle and EchoMind. The paper also attempts to validate its evaluation framework through human–automatic score agreement, human simulator audits, and repeated rollouts.

Critical weaknesses
  1. The novelty of the proposed benchmark and evaluation pipeline is insufficiently distinguished from prior research. MULTI-Bench already investigates interactive multi-turn emotional intelligence in spoken dialogue, while EVA-Bench introduces dynamic audio-based user simulation, simulation-error validation, and end-to-end voice-agent evaluation. Earlier emotional-support studies, including MultiESC and FADO, explicitly investigate evolving user states, feedback-aware strategy selection, and long-term conversational objectives. ParaS2SAlign (ICLR 2026) further establishes GRPO-based optimization of spoken response content and vocal appropriateness. While DynaEmpathy combines these ideas in an empathy-specific framework, the manuscript does not convincingly demonstrate that the integration constitutes a substantial methodological advance over its closest predecessors. More importantly, the central GRPO innovations are not supported by essential ablation studies. The authors propose responsibility-routed semantic rewards and modality-decomposed optimization, yet only compare the base model, SFT, and the complete GRPO pipeline. There is no conventional GRPO baseline with a unified reward, no ablation removing responsibility routing, and no direct comparison establishing the necessity of separating semantic and acoustic optimization. Furthermore, the responsibility router uses four predefined setting/stage categories rather than learning strategy transitions from evolving user states. The GRPO procedure optimizes immediate responses at fixed dialogue states without rolling out their future interaction consequences. Therefore, the experiments do not establish that the model learns a genuinely dynamic, long-horizon empathetic strategy rather than improved context-conditioned response generation.

  2. The benchmark's central claim is that it measures adaptive empathy across evolving spoken interactions. However, I am not convinced that the current evaluation adequately separates this capability from initial affect recognition, general response quality, or conformity to predefined evaluation criteria. First, Appendix C.4 specifies that missing a salient emotional cue in the opening problem-solving response deterministically caps adaptive empathy at 3, preventing the trajectory from passing regardless of subsequent recovery. Although timely emotional attunement is a reasonable desideratum, this scoring rule confounds initial affect recognition with later adaptation. The particularly low problem-solving pass rates may therefore be partly attributable to this hard constraint rather than inadequate multi-turn strategy adaptation. Second, there are no controlled counterfactual experiments that preserve the lexical content and dialogue history while varying only vocal affect. Without such comparisons, it is difficult to establish whether the trained model meaningfully exploits emotional information conveyed by speech or primarily learns improved text-based responses. Third, the benchmark depends entirely on synthetic user scenarios and simulated interactions. Although the authors conduct human simulator audits, inter-annotator agreement is extremely low for several criteria, including response contingency (α = 0.022) and state/disclosure coherence (α = 0.076). The low majority-confirmed serious-issue rate therefore provides limited reassurance about simulation reliability, and behavioral consistency does not establish fidelity to real human reactions. Also, the external evaluations are primarily single-turn benchmarks, while the human trajectory evaluation excludes the SFT variant. Thus, neither provides a complete independent validation of the claimed improvement in dynamic multi-turn empathy.

Questions for authors
  1. What specific technical capabilities distinguish DynaEmpathy from MULTI-Bench and EVA-Bench beyond combining existing interactive simulation and evaluation mechanisms within an empathy-specific setting? Similarly, what is the methodological advance of responsibility-routed, modality-decomposed GRPO over prior paralinguistic speech optimization methods such as ParaS2SAlign?

  2. Can the authors provide controlled ablations comparing the proposed method against (a) conventional GRPO with a unified empathy reward, (b) GRPO without responsibility routing, and (c) GRPO without modality-decomposed optimization, under comparable training data and computational budgets?

  3. How sensitive are the reported results to the deterministic first-turn adaptive-empathy cap in Appendix C.4? Can the authors report results without this constraint and separately quantify initial emotional attunement and subsequent conversational adaptation Additionally, is there evidence that full dialogue history is necessary for successful performance, such as a history-restricted evaluation?

  4. Can the authors provide matched evaluations where the same utterance text and dialogue context are presented with different vocal emotional cues? Do these changes produce appropriate differences in subsequent conversational strategies? How do the authors interpret the extremely low inter-annotator agreement in the simulator audit, and what evidence supports the simulator as a reliable proxy for real emotional interaction?

  5. Table 1 reports a MiniCPM-o-4.5 emotion-disclosure pass rate of 18.4%, which does not appear compatible with an integer number of successful cases out of the stated 300 conditions when rounded to one decimal place. Similarly, OpenS2S reports 5.0% problem-solving and 20.7% emotion-disclosure success but 12.9% overall, whereas the implied counts (15 and 62 out of 300) yield approximately 12.8% overall. Could the authors provide the exact success counts, effective denominators, and aggregation procedure to explain these discrepancies? Furthermore, can the authors report repeated evaluations or paired uncertainty estimates for the Base, SFT, and GRPO variants?

Reviewer_vrMjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work introduces DYNAEMPATHY, a benchmark in which a fixed seed determines the simulated user's initial state and a corresponding user simulator produces plausible feedback from a real user in the current speech-only dialogue turn. It evaluates the trajectory-level empathy of SpeechLMs in conversational interaction, and divides the main empathy scenarios into two categories: implicit-affect problem solving and emotion disclosure. The authors build a pipeline for evaluating empathy in multi-turn dialogue and test existing SpeechLMs on this benchmark. In addition, they use the scenario-grounded seed construction pipeline behind the benchmark to construct data beyond the benchmark, and design a benchmark-aligned GRPO method, which improves the results of Fun-Audio-Chat-8B on both the in-domain multi-turn benchmark and out-of-domain single-turn benchmarks.

Critical strengths

1.The data construction adopts seed construction, which reduces the problem of LLM-generated data containing too many repeated or similar scenarios. And after multiple rounds of both human and automatic quality control, the text quality and audio quality should be well guaranteed. The simulated turns generated during the user simulator's interaction also go through multiple rounds of quality control, which ensures the simulation quality during evaluation and improves the credibility of the evaluation results.

2.Besides the benchmark, the paper also provides a solution with a certain degree of innovation, and the work covers a relatively broad scope.

  1. During interaction, the user simulator is also influenced to some extent by the vocal emotion and cues in the assistant model's responses. This is fairly close to how humans interact, and is relatively novel.
Critical weaknesses

1.Below Table 4 in Appendix A.1, the difference between this work and MULTI-Bench is described as "waveform feedback to MULTI-Bench's user policy and runtime simulator checks are not specified." This distinction is only a simple statement, and I think this comparison may require more detailed justification.

2.The GRPO in this paper is designed with three contributing components — responsibility routing, modality decomposition, and stop-gradient — but there is no corresponding ablation study, making it hard to tell whether each component actually contributes, and how large each one's share of the contribution is.

  1. The GRPO experiments are only conducted on Fun-Audio-Chat-8B, so it is difficult to confirm the generalizability of the method. Experimental results of applying this method to other models (e.g., MiniCPM-o or GLM-4-Voice) need to be added.
Questions for authors
  1. Regarding "two real-world interaction settings: implicit-affect problem solving and emotion disclosure" — what is the source for constructing these two types of scenarios? Is there any theoretical basis for this division?
  2. The paper mentions DYNAEMPATHY (audio demos) in the Introduction, but I could not find the corresponding demos in the supplementary materials. I am curious about the actual data and hope to see a demo.
  3. Regarding the use of Gemini 3.6 Flash for generating the user simulator's next-turn text and Gemini 3.1 Flash TTS Preview as the user simulator's TTS model — I am curious about the reasons for choosing these components for the user simulator. Were these component choices confirmed through any experimental validation?
  4. The paper uses EchoMind's 235 content groups for out-of-domain evaluation, but EchoMind does not have only 235 content groups — it has 1137. I wonder whether the 235 here went through some kind of filtering; if so, I hope this can be clarified in the paper.
  5. Although §3.3 provides operational definitions of the three dimensions, the key "Trajectory-level assistant judge: core instruction" given in Appendix G.3 only briefly lists the names of the metrics to be evaluated in the prompt, without including the specific definitions and detailed criteria of these metrics. Meanwhile, the authors state that only the API wrapper and formatting boilerplate are omitted, which raises the concern that the judge model may not have received the dimension definitions during evaluation. This could easily lead to drift in the model's judging standards. Could the authors clarify whether the complete evaluation prompt contains the dimension definitions from §3.3 and the detailed criteria for each score? If yes, please provide the complete prompt in the appendix; if not, please verify the stability of the scoring results after adding the definitions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.