acceptodds
Under review as a conference paper at ICLR 2027

Bringing Characters into Conversation: Natural and Emotion-Aware Spoken Role-Play

Abstract

Role-play models can reproduce a character's biography yet struggle to sustain a natural conversation. Spoken interaction also requires responses that account for the user's vocal cues and express empathy in a way that fits the persona. We adapt Fun-Audio-Chat-8B through continued supervised fine-tuning (cSFT) on persona-conditioned replies paired with real and synthetic user speech, followed by group relative policy optimization (GRPO). A history-aware reward rubric scores persona fidelity, contextual relevance, semantic and structural repetition, and speech-cue use. Evaluation covers text-to-text (T2T), speech-to-text (S2T), and speech-to-speech (S2S) responses. Expression naturalness (EN) improves by 0.89–1.29 points over the base model across three T2T evaluations and a supplementary S2T test. On an in-house S2S test of 100 sessions, each with 20–50 exchanges about fictional characters' experiences, EN rises from 1.02 to 3.02. GRPO raises mean empathy scores by 0.08–0.52 points over cSFT across four S2T datasets. On the complete VStyle Implicit Empathy test, which evaluates content and vocal delivery jointly, the score increases from 3.73 to 4.33. Human EN means follow the same checkpoint ordering as the model judge. Reward ablations reveal trade-offs between persona consistency and empathy rather than uniform gains from every criterion.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
spoken role-play, speech-language models, persona fidelity, multi-turn dialogue, paralinguistic understanding, emotion-aware dialogue, speech-to-speech generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_n75pOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper post-trains Fun-Audio-Chat-8B using continued SFT followed by GRPO with a history-aware rubric reward. It aims to improve persona expression, multi-turn conversational quality, and emotional responsiveness in spoken role-play, with evaluations across T2T, S2T, and S2S settings.

Critical strengths

The paper addresses a practically relevant problem and evaluates multiple input/output modalities, with training-stage comparisons, reward ablations, and some human evaluation.

Critical weaknesses
  1. The motivation and incremental contribution relative to existing work are insufficiently established. Figure 1 illustrates the targeted problems but does not quantify their prevalence in competitive existing systems. PersonaPlex already studies role-controlled full-duplex speech interaction and evaluates role adherence and dialogue naturalness. The submission does not clearly establish the capability gap relative to such systems or provide direct comparisons supporting advantages in persona expression or emotional responsiveness. Consequently, the empirical gap motivating this work remains insufficiently established.
  2. The methodological contribution is primarily a training recipe, and the distinctive effects of its key designs are not isolated. The approach combines an existing backbone, standard cSFT/GRPO, and a manually weighted rubric reward. Training-stage comparisons and component-removal ablations indicate that some components are useful, but do not isolate the reward model’s access to full dialogue history or compare against a simpler holistic quality reward under matched data and training budgets. It therefore remains unclear how much of the improvement comes from the emphasized history-aware design rather than additional training and stronger quality supervision.
  3. Conversational progress across turns lacks direct validation. EN primarily measures natural persona expression, while CA combines multiple dimensions. The paper does not directly assess redundant repetition, uptake of user corrections, or response updates following new information. Improvements in aggregate scores therefore do not establish substantive conversational progress.
  4. Joint adaptation to vocal affect and persona lacks direct evidence. Persona consistency and empathy are primarily evaluated separately, without establishing whether the same response appropriately uses vocal emotional cues while preserving the assigned character. Removing the persona-consistency reward actually increases empathy in Table 12, further highlighting why separate scores cannot establish the joint objective.
  5. The evaluation lacks sufficient independent validation. Training rewards and persona evaluation use the same LLM judge, leaving open the possibility of adaptation to judge-specific preferences. Human evaluation only supports improved EN agreement, without validating other metrics, as well as conversational progress or joint adaptation to vocal affect and persona. The number of raters, blinding procedure, and inter-rater agreement are also insufficiently documented.
Questions for authors

When transferring this pipeline to another speech foundation model, which data-construction and reward components can be reused directly, and which depend on Fun-Audio-Chat’s model interfaces?

Reviewer_2gf9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper adapts Fun-Audio-Chat-8B via continued SFT and GRPO with a fixed 15-criterion history-aware rubric to improve persona expression and empathy in spoken role-play. It reports EN gains over the base model across T2T/S2T/S2S evaluations, with human EN matching the model-judge ordering.

Critical strengths

S1. Persona quality is judged over the whole session, with CA, CU, CM and EN scored separately, and the scoring prompts and the 15 criterion reward are included. The split pays off: on CharacterEval every system gets CM 5.00, while EN still ranges from 3.51 to 4.61. S2. Empathy is scored by gemini-3.1-pro-preview, a different model from the GRPO reward judge. Under that judge, GRPO beats cSFT on all four S2T sets (+0.08 to +0.52), and VStyle Implicit Empathy goes from 3.73 for the base model to 4.33.

Critical weaknesses

W1. CA, CU, CM and EN are scored by gpt-5.6-luna, the same model that provides the GRPO reward, and human scores cover EN only, so the CA/CU gains may partly reflect this judge's preferences, as Section 8 also acknowledges. Rescoring CA/CU with gemini-3.1-pro-preview, which the paper already uses for empathy, or a small human rating of CA/CU would address this. W2. All comparisons are against the model's own training stages and three external systems, and the reward ablation removes only 3 of the 15 criteria. Without a GRPO run using a simpler reward, such as a single holistic judge score, it is unclear how much of the gain comes from the 15 criterion history-aware rubric itself, which is the paper's main methodological contribution.

Questions for authors

See Critical Weaknesses

Reviewer_RkEKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper adapts Fun-Audio-Chat-8B for spoken role-play through continued supervised fine-tuning and rubric-based GRPO, while keeping speech modules frozen. Training combines real and synthetic conversations with persona and speech-cue annotations. Text and speech evaluations report improved persona expression and empathy, including long-dialogue expression naturalness (EN) gains from 1.02 to 3.02. Training-stage comparisons, reward ablations, and a human EN assessment provide additional evidence.

Critical strengths
  • This paper jointly studies persona consistency, natural expression, and empathy across text and spoken interactions, including long conversations.

  • Training-stage and reward ablations identify useful contributions and trade-offs; human EN scores also support the ordering of training stages.

Critical weaknesses
  • Using the reward model to judge persona quality may favor its own preferences. Human validation covers only EN, so independent support for broader persona gains remains limited.

  • The long-dialogue result relies on session-level EN with fixed user queries. This does not directly establish reduced repetition or sustained quality in adaptive conversations.

Questions for authors
  • Do persona gains persist with an independent judge or blinded human evaluation?

  • Can you report turn-level repetition rates on the long speech-to-speech dialogues?

  • As an additional test, do the gains persist when follow-up queries adapt to the model's replies?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.