acceptodds
Under review as a conference paper at ICLR 2027

CharDuplex: Building Character-Consistent Full-Duplex Spoken Dialogue Models

Abstract

Full-duplex speech models are moving voice interaction beyond conventional turn-taking, yet natural conversation is shaped not only by when an agent speaks, but also by how it behaves as a conversational character. We present CharDuplex, a character-driven full-duplex speech model that combines real-time spoken interaction with persona-conditioned behavior. We first adapt GLM-4-Voice to an always-on dual-stream architecture and train the model for full-duplex conversation. Then a fully automated pipeline constructs character-conditioned dialogue data from open-source character descriptions for character-conditioned supervised fine-tuning. The model is further refined with the proposed FDGym, where an LLM-simulated user dynamically interacts with the model, enabling reinforcement learning over evolving multi-turn interactions. On SpeechRole-Eval, CharDuplex achieves the highest average score among the evaluated open-source models, while remaining competitive with closed-source systems. It also demonstrates competitive general speech intelligence and strong full-duplex interaction capabilities. CharDuplex demonstrates a practical training recipe for building full-duplex voice assistants that are not only interactive, but also character-consistent.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Full-Duplex Spoken Dialogue, Speech Language Models, Persona Modeling, Role-Playing, Interactive Post-Training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5fMLOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses character consistency in full-duplex models. It presents CharDuplex, a full-duplex model that achieves improved character consistency. The model is trained via SFT followed by RL. Another contribution of this paper is FDGym, an interactive RL framework for full-duplex training. It uses an LLM to simulate user behavior, ensuring that the user's speech depends on the conversational context. The results show that CharDuplex achieves better character consistency than the baselines while maintaining competitive performance on speech intelligence benchmarks.

Objectively speaking, this paper has its merits, but the current version lacks many critical details and discussions. I would be willing to raise my assessment if these issues were adequately addressed.

Critical strengths
  • The paper is generally well-written and easy to follow.
  • The FDGym framework is a meaningful contribution to the field.
  • The results demonstrate the effectiveness of the proposed method.
Critical weaknesses
  • Positioning & Scope: Character consistency is not a problem unique to full-duplex voice assistants. Half-duplex models also need to address this issue. Therefore, I believe the most significant contribution of this paper is its interactive RL framework, which should be applicable to a broader range of problems beyond character consistency. The paper would make a more significant contribution if it demonstrated how RL could address challenges more specific to full-duplex scenarios, such as turn-taking.

  • Unclear Backbone Motivation & Lack of Related Work Discussion: The way GLM-4-Voice is used in this paper is not intuitive. GLM-4-Voice is originally a speech-to-speech model, but the authors retrain it as a speech-to-text model and append a speech synthesis module after the LLM. This differs from TurnGuide's approach [1], which preserves GLM-4-Voice's speech-to-speech capabilities when adapting it to full-duplex interactions. Therefore, the authors should provide a clear comparison between the training methods of CharDuplex and TurnGuide and explain the rationale behind CharDuplex's design choices in the paper.

  • Critical Details Missing:

    • FDGym uses a fixed simulation tick of 160 ms. Does this mean that, every 160 ms, the user simulator LLM needs to decide whether to generate the user's next speech segment and whether to interrupt the assistant? This seems to be a highly resource-intensive process that could significantly slow down training.
    • The details of the speech synthesis module are almost completely missing. What exactly is the architecture of this module? What are its inputs? Does it take only text tokens, or does it also incorporate the LLM's hidden states? What training data are used for this module? What is its streaming configuration? What is the text-speech interleaving ratio? Is any text lookahead used?
    • SpeechRole-Eval appears to be a half-duplex benchmark. How do the authors adapt it to evaluate full-duplex models?
  • Inconsistent Baselines: For example, both Table 1 and 2 include Personaplex, but Table 3 does not have this backbone. Personaplex has released their Full Duplex Bench results in their original paper.

[1] TurnGuide: Enhancing Meaningful Full Duplex Spoken Interactions via Dynamic Turn-Level Text-Speech Interleaving. Interspeech Long 2026.

Questions for authors
  • Why is the LLM trained to produce the EOS token at the end of the assistant's speech? Since the LLM cannot hear the assistant's ongoing speech (it only has access to the generated text tokens), it does not seem to have sufficient information to learn when to output EOS. Therefore, I suspect that the model essentially has to guess when to produce the EOS token.
  • In Table 2, CharDuplex outperforms GLM-4-Voice on many QA benchmarks. This is counterintuitive, as full-duplex training typically degrades performance on QA benchmarks. Is there a reasonable explanation for this improvement?
8 comments
AuthorsOpenReview ↗

Author Response and Additional Experimental Results for Reviewer 5fML

We thank the reviewer for the thoughtful and constructive feedback, and for recognizing the contributions of FDGym and the effectiveness of our approach. We appreciate the detailed technical questions and suggestions, which help us clarify the motivation, implementation, and evaluation of our method. We address each concern below.

W1: Positioning and scope.

We agree that character consistency is not unique to full-duplex models. Our central methodological contribution is FDGym, an interactive RL framework for full-duplex dialogue, with character consistency as the application studied in this paper. Importantly, FDGym explicitly addresses full-duplex-specific interaction behaviors rather than optimizing character consistency alone. We provide two complementary experimental results.

(1) Timing rewards are essential for full-duplex interaction. FDGym incorporates two timing rewards: r_resp for response initiation and r_yield for yielding under user interruption. Removing both rewards leaves character performance nearly unchanged (SpeechRole-Eval Avg: 0.784 vs. 0.785), but substantially degrades full-duplex interaction:

ModelSpeechRole-Eval Avg ↑Interrupt TOR ↑Interrupt latency ↓Backchannel TOR ↓Backchannel JSD ↓
CharDuplex0.7850.9900.5650.7090.889
w/o timing rewards0.7840.4401.4330.8360.979

Without timing rewards, the model produces no responses on the Candor subset, and its interruption TOR drops from 0.990 to 0.440. These results demonstrate that content-based rewards alone are insufficient to maintain full-duplex interaction behavior during RL, and that explicit timing optimization is critical.

(2) Response-dependent interaction improves multi-turn RL. We compare FDGym against two fixed-user RL baselines. In particular, Fixed-all uses the same initialization, rewards, GRPO settings, datasets and optimization over all assistant turns, but removes the interactive closed loop by keeping subsequent user utterances fixed. Fixed-last additionally uses ground-truth assistant prefixes and optimizes only the final response.

ModelIACCPeCKCAvg
CharDuplex-SFT0.8070.8000.7290.7700.777
RL, Fixed-all0.7870.6960.6450.6950.706
RL, Fixed-last0.8380.7760.6850.7580.765
RL, FDGym0.8150.8050.7380.7810.785

Both fixed-user baselines fall below SFT, while FDGym improves all four character-related metrics. In particular, the comparison with Fixed-all supports the importance of response-dependent user continuations for effective multi-turn RL.

Together, these results establish two important aspects of FDGym: explicit optimization of full-duplex interaction timing and response-dependent closed-loop training.

W2: Backbone motivation and comparison with TurnGuide.

Thank you for pointing us to TurnGuide. We clarify our design motivation and the differences below.

Motivation. Our architectural design follows prior work rather than constituting a new contribution. Specifically, we adopt the dual-stream formulation of Chronological Thinking [1] and the decoupled speech generation design of FLAIR [2]. Neither work has published codes or weights. We implement this architecture on the open-source GLM-4-Voice backbone, retaining its pretrained speech tokenizer and LLM while adapting the output stream to predict text and turn-control tokens.

Although GLM-4-Voice originally supports speech-to-speech generation, we deliberately decouple language modeling from acoustic generation. This allows the LLM to focus on response content and temporal interaction control, which stays close to text language modeling, while a separate module handles speech synthesis. The resulting system remains speech-to-speech, but the LLM no longer needs to generate acoustic tokens directly.

Comparison with TurnGuide. Both methods adapt GLM-4-Voice to full-duplex interaction, but differ in their modeling approaches:

TurnGuideCharDuplex
Stream structureInterleaved user and assistant chunksTwo synchronous streams with framewise embedding fusion
LLM outputInterleaved text and speech tokensText and control tokens (PAD/BOS/EOS)
Speech generationSpeech tokens predicted by the LLMSeparate module conditioned on text tokens and LLM hidden states
Temporal controlChunk-based text–speech interleavingFrame-level turn control with turn-level text–speech alignment

We do not claim this architectural choice as a novel contribution or establish its superiority over TurnGuide. Our listed contributions (Section 1) do not include these architectural designs. Rather, it provides the backbone for our character-conditioned training pipeline and FDGym. We have clarified this motivation and add the TurnGuide comparison in the revised paper.

AuthorsOpenReview ↗

Author Response and Additional Experimental Results for Reviewer 5fML

W3: Missing details.

aa Cost of the 160 ms tick. The simulator makes a decision every tick, but it queries the LLM on only a few of them. Ticks are needed because the two sides share one timeline: the simulator reacts to the response text that has been spoken so far, which is exposed gradually at the speaking rate, not to the text the model has already generated. At each tick the simulator applies simple rules: if a user utterance is being played, it streams the next 160 ms; otherwise it waits. The LLM is called only on two events: when the spoken response reaches a sentence boundary (to check for an interruption), and when the assistant has been silent for a waiting period (to generate the next user utterance, followed by TTS). The simulation clock is paused during LLM and TTS calls, so their latency does not affect the dialogue timing. A trajectory therefore needs only a few LLM calls. We have added this in Appendix C.

bb Speech generation module. It follows the design of FLAIR [2]. We have added the configurations in Appendix C.

  • Architecture: an autoregressive Transformer decoder (4 layers, hidden size 1024, 8 heads) that predicts CosyVoice2 speech tokens, with a 2-layer causal encoder over its conditioning inputs.

  • Inputs: both the response text tokens and the LLM hidden states.

  • Streaming: the module runs frame by frame with the LLM. For each 80 ms frame, it predicts two CosyVoice2 tokens, which the CosyVoice2 streaming flow-matching decoder converts to waveform.

  • Training: The LLM is frozen while the speech generation module is trained separately. The training data is the same as the general full-duplex SFT and character-conditioned SFT.

  • Interleaving ratio: there is none. Text and speech tokens are not interleaved in one sequence; text is predicted by the LLM and speech by this module, aligned on the same frame timeline.

  • Lookahead: At each frame, the module attends to all text tokens generated so far, acts like an LLM with KV-cache. Text tokens are produced at one per frame, faster than they are spoken, so when the module generates the speech for one word, the text of the following words is usually already available.

  • Latency: the module itself adds no delay; it emits two speech tokens per frame. The speech latency is determined by the CosyVoice2 streaming flow-matching decoder, which generates audio in chunks of 15 speech tokens with a 3-token lookahead.

cc SpeechRole-Eval in full-duplex. We convert each dialogue into a continuous audio stream: the user utterances are concatenated in order, with a silence gapng enough for the model to respond\ (no less than 30s). Thecharacter description is the system prompt. The model runs in full-duplex mode over the whole stream, and its response in each gap is scored by the official Gemini otocol.

We have added all of these details to the paper.

W4: Inconsistent baselines.

Thank you for pointing this out. We have added PersonaPlex to Table 3, using the Full-Duplex-Bench results reported in its original paper.

Q1: Why does the LLM predict <EOS> at the end of speech?

<EOS> is a speech-control token rather than a prediction made after observing the generated waveform. The LLM and speech generation module operate synchronously on a shared 80 ms frame timeline. At each frame, the LLM predicts a text or control token, while the separate speech module generates two CosyVoice2 speech tokens per frame during speech output. Thus, the LLM controls when speech starts and stops through <BOS> and <EOS>, without needing to listen to its own generated speech.

During SFT, <BOS> and <EOS> are aligned with the onset and offset of the reference assistant speech. After generating the response text, the LLM continues predicting <PAD> tokens until the speech ends, providing explicit frame-level supervision for speech duration.

  • Normal completion. The LLM knows the text of the turn and, since it runs on a fixed 80 ms frame clock, how long it has been speaking. From the training data it learns how long a given text takes to speak, and emits <EOS> accordingly.

  • Interruption: The LLM continuously receives incoming user audio. When the user interrupts, the training target places <EOS> at the point where the assistant yields, allowing the model to learn interruption-triggered speech termination.

Therefore, the LLM does not need to observe the synthesized waveform to predict <EOS>; it learns a frame-level speech-control policy from supervised text–speech timing alignment. We have clarified this mechanism in Section 3.2.

AuthorsOpenReview ↗

Author Response and Additional Experimental Results for Reviewer 5fML

Q2: QA results compared with GLM-4-Voice.

Your intuition holds when a model is adapted to full-duplex with conversational data only. That is not our setting. CharDuplex is not GLM-4-Voice with a light full-duplex fine-tuning: its first training stage uses about 620K hours of data, of which about 90K hours are spoken QA 70KhoursofinstructionQAand20KhoursofASR−QA70K hours of instruction QA and 20K hours of ASR-QA. The model therefore learns QA together with full-duplex interaction, on data that GLM-4-Voice was not trained on. In addition, the LLM predicts only text and control tokens, which makes its modeling task simpler than generating text and speech tokens together seeW2see W2. We have added this explanation to Section 5.

[1] Wu, D., Zhang, H., Chen, C., Zhang, T., Tian, F., Yang, X., ... & Chng, E. S. (2026, August). Chronological thinking in full-duplex spoken dialogue language models. In Proceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue pp\. 473-485.

[2] Wu, D., Zhang, T., Li, Y., Liu, H., Chen, C., Chng, E. S., & Bengio, Y. 20262026. The silent thought: Modeling internal cognition in full-duplex spoken dialogue models via latent reasoning. arXiv preprint arXiv:2603.17837.

Reviewer_ZyREOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CharDuplex combines a character-aware full-duplex speech model with a post-training pipeline, including reinforcement learning in FDGym. It achieves competitive results on VoiceBench, Full-Duplex-Bench, and SpeechRole-Eval.

Critical strengths

The character-aware architecture and post-training pipeline form a useful approach to role-based full-duplex conversation.

The model performs well across general speech, full-duplex interaction, and role consistency, rather than only on one benchmark.

Critical weaknesses

The RL gains are small. Without clearer evidence of their reliability or a controlled comparison, I am not convinced that FDGym’s interactive design adds much beyond the rest of the training pipeline. This is my main concern, since FDGym is a central contribution.

The persona evaluation mainly covers response content. It does not show whether characters differ consistently in interaction style, such as interruption, backchanneling, or response timing. If the intended scope is content-level consistency only, the paper should make that clear.

Questions for authors

Do any existing runs or ablations show that the FDGym gains are reliable and specifically come from its interactive closed loop, rather than RL alone?

Is the character modeling limited to response content, or does it also cover vocal and interaction style? If the latter, what existing evidence supports it?

3 comments
AuthorsOpenReview ↗

Author Response and Additional Experimental Results for Reviewer ZyRE

We sincerely thank the reviewer for the constructive feedback and for highlighting the importance of isolating the contribution of interactive closed-loop training.

Q1: Are the FDGym gains reliable, and do they come from the interactive closed loop?

To investigate the contribution of closed-loop interaction, we compare FDGym with two RL baselines using fixed user tracks from the SFT data. All three RL variants share the same initialization, reward components, and GRPO settings:

  • Fixed-all: The prerecorded user track remains unchanged regardless of the model's generated responses, and all assistant turns are optimized.
  • Fixed-last: Prerecorded user tracks and ground-truth assistant prefixes are used, with optimization restricted to the final assistant response.
ModelIACCPeCKCAvg
CharDuplex-SFT0.8070.8000.7290.7700.777
RL, Fixed-all0.7870.6960.6450.6950.706
RL, Fixed-last0.8380.7760.6850.7580.765
RL, FDGym0.8150.8050.7380.7810.785

The results highlight the importance of closed-loop interaction. Both fixed-user RL baselines underperform SFT, particularly on personality and knowledge consistency, whereas FDGym further improves all four character-related metrics. Importantly, Fixed-all provides a direct open-loop comparison with FDGym: both optimize all assistant turns under identical RL settings, with the key difference being whether subsequent user behavior adapts to the model's actual responses.

Why does Fixed-all perform substantially worse? The fundamental issue is that fixed user trajectories break the causal relationship between the model's responses and subsequent user behavior, introducing two forms of inconsistency:

  • Semantic inconsistency: Subsequent user utterances follow the original SFT dialogue rather than the model's generated responses. For example, the user may ask follow-up questions about an explanation the model never provided, refer to information that was never mentioned, or continue discussing a topic that the model has already redirected. Conversely, if the model asks a clarification question or introduces new information, the prerecorded user cannot respond accordingly. These inconsistencies accumulate across turns, progressively disconnecting the dialogue from the model's actual behavior.

  • Temporal inconsistency: The prerecorded user speech retains its original timing, while the model may generate responses with different durations and speaking patterns. Consequently, an interruption originally intended to occur during an assistant response may arrive before the model starts speaking or after it has already finished. Similarly, a backchannel may occur at an inappropriate conversational moment. These mismatches disrupt the temporal coordination between listening and speaking that full-duplex models are trained to maintain.

Crucially, these inconsistencies directly affect RL optimization. GRPO assigns trajectory-level rewards to the model's generated tokens, but the resulting user behavior in Fixed-all is not causally determined by those tokens. Consequently, the model may be rewarded or penalized for conversational outcomes that would not arise from its actual responses, leading to misleading credit assignment and potentially reinforcing inappropriate conversational behaviors.

This mechanism also helps explain why Fixed-last performs better than Fixed-all (0.765 vs. 0.706): ground-truth assistant prefixes reduce accumulated history divergence. However, Fixed-last also restricts optimization to the final assistant response, rather than optimizing decisions throughout a multi-turn interaction. It therefore does not address the challenge of learning from the consequences of the model's own behavior across multiple turns.

FDGym addresses this challenge by preserving the causal relationship between model responses and subsequent user reactions throughout the rollout. The controlled comparison with Fixed-all therefore provides evidence that response-dependent interaction is an important component of effective multi-turn RL, rather than merely applying RL to fixed dialogue trajectories.

We have incorporated these comparisons into Appendix G of the revised manuscript.

AuthorsOpenReview ↗

Author Response and Additional Experimental Results for Reviewer ZyRE

Q1: Are the FDGym gains reliable, and do they come from the interactive closed loop?

To further assess the reliability of the improvement over SFT, we conducted paired statistical analyses on SpeechRole-Eval using the official Gemini evaluator and four additional LLM judges. The results are summarized below.

Evaluation JudgeSFTFDGymImprovement95% CIp-value
Gemini 2.5 Pro0.7770.785+0.008[0.0006, 0.0184]0.0386
GPT-6-sol0.8250.837+0.012[0.0024, 0.0218]0.0151
Claude Sonnet 5.50.7040.713+0.001[0.0001, 0.0197]0.0488
Claude Opus 5.50.7290.739+0.010[0.0001, 0.0205]0.0468
Kimi K3.1 Pro0.7900.804+0.014[0.0006, 0.0272]0.0412

All five judges consistently report positive improvements over SFT, with 95% confidence intervals excluding zero and p-values below 0.05. These results provide additional statistical evidence for the improvement over SFT across different evaluators, complementing our controlled RL comparisons that demonstrate the contribution of response-dependent closed-loop training.

Q2: Does character modeling cover vocal and interaction style?

Our character modeling focuses on response content and linguistic persona, including character identity, personality, knowledge, and role-specific constraints. These are the dimensions evaluated by SpeechRole-Eval through Instruction Adherence, Conversational Coherence, Personality Consistency, and Knowledge Consistency.

In this paper, response timing, interruption handling, and backchanneling are modeled as general full-duplex capabilities shared across characters, rather than character-specific interaction styles. This distinction is reflected in our training: character descriptions condition the model's responses, while FDGym's timing rewards help maintain appropriate interaction behavior regardless of the assigned character. Thus, our approach combines character-conditioned response generation with general full-duplex interaction, without explicitly modeling character-dependent vocal or interaction styles.

We have clarified this scope in the revised Introduction. Extending character conditioning to vocal characteristics and interaction styles, such as character-specific speaking patterns and interruption preferences, is a natural direction for future work.

AuthorsOpenReview ↗

Follow-up on Rebuttal and Additional Experimental Evidence

We sincerely appreciate the reviewer's thoughtful and constructive feedback.

We have carefully addressed your concerns through additional controlled RL comparisons, statistical significance analysis, and evaluations across multiple independent LLM judges. We have also clarified the scope of character modeling in the revised manuscript.

As the discussion period is approaching its end, we would sincerely appreciate it if you could review our additional experimental evidence and reconsider your assessment in light of these results. We would be happy to address any remaining concerns or provide further clarification.

Thank you very much for your time and consideration.

Reviewer_nsWdOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CharDuplex adapts GLM-4-Voice for character-conditioned full-duplex dialogue. It combines automated dialogue construction and verification, supervised fine-tuning, and interactive RL in FDGym. It achieves the highest SpeechRole-Eval average among the evaluated open-source models and competitive results on selected speech and interaction metrics.

Critical strengths
  1. The data pipeline is carefully designed, including grounding checks, dialogue repair, speech-quality control, and interruption-aware construction. Figure 1 illustrates it clearly.

  2. The evaluation covers character consistency, general speech intelligence, and full-duplex interaction. The SFT model already performs strongly, supporting the usefulness of the supervised training recipe.

Critical weaknesses
  1. The benefits from RL is marginal. According to Table 1, the average gain on SpeechRole-Eval increases from 0.777 to 0.785, without uncertainty estimates. No matched fixed-user-continuation RL or additional-SFT comparison isolates the benefit of interactive closed-loop training.

  2. The rewards are defined, but insufficiently validated. The reliability of the LLM scores is not established, and the timing-reward ablation lacks numerical results. Timing rewards also use simulated time and estimated speaking rates while model-side waveform generation is disabled.

  3. Architectural novelty needs clearer positioning. Dual streams, embedding summation, and separate speech generation closely resemble cited prior work, particularly Chronological Thinking. The paper should distinguish inherited components from new choices.

  4. Interaction preservation needs qualification. Backchannel turn-over rate worsens from 0.636 to 0.709, where lower is better. Separate character and interaction benchmarks also leave character consistency under interruptions insufficiently demonstrated.

Questions for authors
  1. Do existing runs establish the reliability of the RL gains? Are matched fixed-user-continuation RL or additional-SFT comparisons available?

  2. For the two Qwen-235B-judged rewards, do existing checks quantify score variability when the same trajectory is evaluated repeatedly, or agreement with another judge or human assessment?

  3. What are the exact result on the SpeechRole-Eval and Full-Duplex-Bench for the Appendix G variant that removes both timing rewards.

  4. Compared with previous work, how does CharDuplex differ in framewise embedding fusion, PAD/BOS/EOS turn control, text–speech alignment, and speech generation? Please identify which components are inherited and which changes are specific to adapting GLM-4-Voice.

  5. What explains the backchannel regression? Were interaction evaluations conducted with character prompts enabled?

7 comments
AuthorsOpenReview ↗

Author Response and Additional Experimental Results for Reviewer nsWd

We thank the reviewer for the constructive comments and for recognizing our data pipeline and the breadth of our evaluation. We clarify each question below.

Q1: Reliability of RL gains and fixed-user-continuation baseline.

To isolate the contribution of interactive closed-loop training, we compare FDGym with two RL baselines using fixed user tracks from the SFT data. All three RL variants share the same initialization, reward components, and GRPO settings:

  • Fixed-all: the user track does not adapt to model responses. We directly use the user track of SFT data, regardless of what the LLM replies; all turns are optimized.

  • Fixed-last: fixed user tracks and ground-truth assistant prefixes are used; only the response of the last turn is optimized.

ModelIACCPeCKCAvg
CharDuplex-SFT0.8070.8000.7290.7700.777
RL, Fixed-all0.7870.6960.6450.6950.706
RL, Fixed-last0.8380.7760.6850.7580.765
RL, FDGym0.8150.8050.7380.7810.785

FDGym is the only RL variant that improves over SFT across all four metrics, whereas both fixed-user baselines underperform SFT, particularly on personality and knowledge consistency.

The key distinction is that Fixed-all breaks the causal relationship between model responses and subsequent user behavior. Fixed user tracks can introduce semantic inconsistencies in dialogue content and temporal mismatches in interruptions and backchannels. For example, the user may ask follow-up questions about an explanation the model never provided, refer to information that was never mentioned, or continue discussing a topic that the model has already redirected. These inconsistencies affect trajectory-level reward assignment, as the resulting outcomes may not reflect the consequences of the model's actual responses. FDGym avoids these mismatches through response-dependent user simulation.

Fixed-last reduces accumulated history inconsistency through ground-truth prefixes, but optimizes only the final turn rather than the complete multi-turn interaction. Together, these comparisons support the importance of closed-loop interaction for effective multi-turn RL.

We have incorporated these results into Appendix G.

Q2: Reliability of the LLM-judged rewards.

Our setup addresses reward variability and judge-specific bias through three complementary mechanisms:

  • Independent evaluation. FDGym uses Qwen-235B to compute training rewards, whereas SpeechRole-Eval uses Gemini 2.5 Pro following its official evaluation protocol. The observed performance improvements are therefore measured independently of the reward model used during training, reducing the risk that the reported gains merely reflect exploitation of the training judge.

  • Group-relative reward normalization: GRPO computes advantages by standardizing rewards across trajectories sharing the same initial dialogue condition. This removes common additive scoring offsets within each group and reduces dependence on absolute reward calibration. The optimization therefore focuses on relative differences between sampled trajectories rather than their raw reward magnitudes.

  • Deterministic judging protocol: Both reward judges use greedy decoding, fixed evaluation criteria, and structured JSON outputs to reduce sampling-induced variability. Separate prompts are used for interaction quality and character consistency, allowing the two reward components to assess distinct aspects of the dialogue.

These measures serve different purposes: deterministic evaluation reduces sampling variability, group-relative normalization mitigates common scoring offsets, and independent downstream evaluation provides evidence that the learned improvements transfer beyond the training judge.

Q3: Timing-reward ablation.

We conducted an ablation removing both deterministic timing rewards, while retaining the LLM-judged interaction and character rewards.

SpeechRole-Eval

ModelIACCPeCKCAvg
CharDuplex0.8150.8050.7380.7810.785
w/o timing rewards0.8650.7880.7150.7680.784

Full-Duplex-Bench

ModelInterrupt TOR ↑Interrupt latency ↓Backchannel TOR ↓Backchannel JSD ↓
CharDuplex0.9900.5650.7090.889
w/o timing rewards0.4401.4330.8360.979

Removing the timing rewards gives no gain in character performance and degrades full-duplex interaction. The model fails to respond on all Candor samples, so pause handling and smooth turn-taking cannot be computed, and it resumes after only 44% of user interruptions. It also takes turns more often during user backchannels. The LLM judges score what the model says but do not constrain when it speaks, so timing drifts without r_resp and r_yield. We have added these results in Appendix G.

AuthorsOpenReview ↗

Author Response and Additional Experimental Results for Revierwer nsWd

Q4: Compared with previous work

We agree that dual streams, embedding summation, and separate speech generation follow prior full-duplex work, Chronological Thinking [1] and FLAIR [2]. Neither work has published codes or weights. We implement this architecture on the open-source GLM-4-Voice backbone. Our listed contributions (Section 1) do not include these architectural designs. GLM-4-Voice-specific changes include:

  • user speech uses GLM-4-Voice's native WhisperVQ tokens, so both streams share the pretrained embedding table without an extra encoder;

  • BOS/EOS/PAD reuse existing GLM special tokens rather than new vocabulary;

This adaptation is an enabling step rather than our main contribution. It provides the full-duplex backbone on which our character-conditioned data pipeline and FDGym are built and evaluated. We have revised the descriptions to make it clear. We will release the code and model weights, providing a reproducible full-duplex model on an open backbone.

Q5: Backchannel regression and evaluation setting.

We acknowledge the regression in Backchannel TOR (0.636→0.709), indicating that the model takes the floor more often during user backchannels. Other backchannel metrics remain nearly unchanged, while Smooth Turn-Taking TOR improves (0.840→0.857) and Pause Handling TOR remains stable. These results suggest a trade-off between turn initiation and avoiding unnecessary responses to backchannels, rather than a general degradation of full-duplex interaction. One possible explanation is that FDGym encourages timely response initiation and interruption yielding, without explicitly optimizing the suppression of unnecessary responses to backchannels.

Full-Duplex-Bench was evaluated without character prompts, following its standard protocol, to assess whether character-oriented training preserves general interaction capabilities. Character consistency and interruption handling are trained within the same conversational contexts. Character-conditioned SFT includes interruptions, backchannels, and pauses under character prompts, while FDGym optimizes character consistency and interruption yielding within the same interactive rollouts. Although the current benchmarks evaluate these capabilities separately, joint evaluation of character consistency under interruptions remains a valuable direction for future work.

[1] Wu, D., Zhang, H., Chen, C., Zhang, T., Tian, F., Yang, X., ... & Chng, E. S. (2026, August). Chronological thinking in full-duplex spoken dialogue language models. In Proceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue pp\. 473-485.

[2] Wu, D., Zhang, T., Li, Y., Liu, H., Chen, C., Chng, E. S., & Bengio, Y. 20262026. The silent thought: Modeling internal cognition in full-duplex spoken dialogue models via latent reasoning. arXiv preprint arXiv:2603.17837.

I appreciate the authors for providing the response.

Q1: It is great to see that the proposed RLGym outperforms replayed continuations within their RL setup. However, it is still unclear whether the 0.008 average improvement over the SFT baseline is statistically significant.

Q2: I acknowledged the mentioned mechanisms such as independent evaluation. However, any quantitative cross-judge or human-to-LLM agreement will still be helpful.

Q3: The ablation on timing reward addressed my request.

Q4: This clarification addresses my questions.

Q5: Thank you for clarifying that Full-Duplex-Bench was evaluated without character prompts. The setting is now clear. However, this leaves two questions open: whether interaction quality is preserved when character prompts are enabled, and whether the model stays in character when interrupted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.