- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses character consistency in full-duplex models. It presents CharDuplex, a full-duplex model that achieves improved character consistency. The model is trained via SFT followed by RL. Another contribution of this paper is FDGym, an interactive RL framework for full-duplex training. It uses an LLM to simulate user behavior, ensuring that the user's speech depends on the conversational context. The results show that CharDuplex achieves better character consistency than the baselines while maintaining competitive performance on speech intelligence benchmarks.
Objectively speaking, this paper has its merits, but the current version lacks many critical details and discussions. I would be willing to raise my assessment if these issues were adequately addressed.
Critical strengths
- The paper is generally well-written and easy to follow.
- The FDGym framework is a meaningful contribution to the field.
- The results demonstrate the effectiveness of the proposed method.
Critical weaknesses
-
Positioning & Scope: Character consistency is not a problem unique to full-duplex voice assistants. Half-duplex models also need to address this issue. Therefore, I believe the most significant contribution of this paper is its interactive RL framework, which should be applicable to a broader range of problems beyond character consistency. The paper would make a more significant contribution if it demonstrated how RL could address challenges more specific to full-duplex scenarios, such as turn-taking.
-
Unclear Backbone Motivation & Lack of Related Work Discussion: The way GLM-4-Voice is used in this paper is not intuitive. GLM-4-Voice is originally a speech-to-speech model, but the authors retrain it as a speech-to-text model and append a speech synthesis module after the LLM. This differs from TurnGuide's approach [1], which preserves GLM-4-Voice's speech-to-speech capabilities when adapting it to full-duplex interactions. Therefore, the authors should provide a clear comparison between the training methods of CharDuplex and TurnGuide and explain the rationale behind CharDuplex's design choices in the paper.
-
Critical Details Missing:
- FDGym uses a fixed simulation tick of 160 ms. Does this mean that, every 160 ms, the user simulator LLM needs to decide whether to generate the user's next speech segment and whether to interrupt the assistant? This seems to be a highly resource-intensive process that could significantly slow down training.
- The details of the speech synthesis module are almost completely missing. What exactly is the architecture of this module? What are its inputs? Does it take only text tokens, or does it also incorporate the LLM's hidden states? What training data are used for this module? What is its streaming configuration? What is the text-speech interleaving ratio? Is any text lookahead used?
- SpeechRole-Eval appears to be a half-duplex benchmark. How do the authors adapt it to evaluate full-duplex models?
-
Inconsistent Baselines: For example, both Table 1 and 2 include Personaplex, but Table 3 does not have this backbone. Personaplex has released their Full Duplex Bench results in their original paper.
[1] TurnGuide: Enhancing Meaningful Full Duplex Spoken Interactions via Dynamic Turn-Level Text-Speech Interleaving. Interspeech Long 2026.
Questions for authors
- Why is the LLM trained to produce the EOS token at the end of the assistant's speech? Since the LLM cannot hear the assistant's ongoing speech (it only has access to the generated text tokens), it does not seem to have sufficient information to learn when to output EOS. Therefore, I suspect that the model essentially has to guess when to produce the EOS token.
- In Table 2, CharDuplex outperforms GLM-4-Voice on many QA benchmarks. This is counterintuitive, as full-duplex training typically degrades performance on QA benchmarks. Is there a reasonable explanation for this improvement?
8 comments
Author Response and Additional Experimental Results for Reviewer 5fML
W3: Missing details.
Cost of the 160 ms tick. The simulator makes a decision every tick, but it queries the LLM on only a few of them. Ticks are needed because the two sides share one timeline: the simulator reacts to the response text that has been spoken so far, which is exposed gradually at the speaking rate, not to the text the model has already generated. At each tick the simulator applies simple rules: if a user utterance is being played, it streams the next 160 ms; otherwise it waits. The LLM is called only on two events: when the spoken response reaches a sentence boundary (to check for an interruption), and when the assistant has been silent for a waiting period (to generate the next user utterance, followed by TTS). The simulation clock is paused during LLM and TTS calls, so their latency does not affect the dialogue timing. A trajectory therefore needs only a few LLM calls. We have added this in Appendix C.
Speech generation module. It follows the design of FLAIR [2]. We have added the configurations in Appendix C.
-
Architecture: an autoregressive Transformer decoder (4 layers, hidden size 1024, 8 heads) that predicts CosyVoice2 speech tokens, with a 2-layer causal encoder over its conditioning inputs.
-
Inputs: both the response text tokens and the LLM hidden states.
-
Streaming: the module runs frame by frame with the LLM. For each 80 ms frame, it predicts two CosyVoice2 tokens, which the CosyVoice2 streaming flow-matching decoder converts to waveform.
-
Training: The LLM is frozen while the speech generation module is trained separately. The training data is the same as the general full-duplex SFT and character-conditioned SFT.
-
Interleaving ratio: there is none. Text and speech tokens are not interleaved in one sequence; text is predicted by the LLM and speech by this module, aligned on the same frame timeline.
-
Lookahead: At each frame, the module attends to all text tokens generated so far, acts like an LLM with KV-cache. Text tokens are produced at one per frame, faster than they are spoken, so when the module generates the speech for one word, the text of the following words is usually already available.
-
Latency: the module itself adds no delay; it emits two speech tokens per frame. The speech latency is determined by the CosyVoice2 streaming flow-matching decoder, which generates audio in chunks of 15 speech tokens with a 3-token lookahead.
SpeechRole-Eval in full-duplex. We convert each dialogue into a continuous audio stream: the user utterances are concatenated in order, with a silence gapng enough for the model to respond\ (no less than 30s). Thecharacter description is the system prompt. The model runs in full-duplex mode over the whole stream, and its response in each gap is scored by the official Gemini otocol.
We have added all of these details to the paper.
W4: Inconsistent baselines.
Thank you for pointing this out. We have added PersonaPlex to Table 3, using the Full-Duplex-Bench results reported in its original paper.
Q1: Why does the LLM predict <EOS> at the end of speech?
<EOS> is a speech-control token rather than a prediction made after observing the generated waveform. The LLM and speech generation module operate synchronously on a shared 80 ms frame timeline. At each frame, the LLM predicts a text or control token, while the separate speech module generates two CosyVoice2 speech tokens per frame during speech output. Thus, the LLM controls when speech starts and stops through <BOS> and <EOS>, without needing to listen to its own generated speech.
During SFT, <BOS> and <EOS> are aligned with the onset and offset of the reference assistant speech. After generating the response text, the LLM continues predicting <PAD> tokens until the speech ends, providing explicit frame-level supervision for speech duration.
-
Normal completion. The LLM knows the text of the turn and, since it runs on a fixed 80 ms frame clock, how long it has been speaking. From the training data it learns how long a given text takes to speak, and emits
<EOS>accordingly. -
Interruption: The LLM continuously receives incoming user audio. When the user interrupts, the training target places
<EOS>at the point where the assistant yields, allowing the model to learn interruption-triggered speech termination.
Therefore, the LLM does not need to observe the synthesized waveform to predict <EOS>; it learns a frame-level speech-control policy from supervised text–speech timing alignment. We have clarified this mechanism in Section 3.2.
Author Response and Additional Experimental Results for Reviewer 5fML
Q2: QA results compared with GLM-4-Voice.
Your intuition holds when a model is adapted to full-duplex with conversational data only. That is not our setting. CharDuplex is not GLM-4-Voice with a light full-duplex fine-tuning: its first training stage uses about 620K hours of data, of which about 90K hours are spoken QA . The model therefore learns QA together with full-duplex interaction, on data that GLM-4-Voice was not trained on. In addition, the LLM predicts only text and control tokens, which makes its modeling task simpler than generating text and speech tokens together . We have added this explanation to Section 5.
[1] Wu, D., Zhang, H., Chen, C., Zhang, T., Tian, F., Yang, X., ... & Chng, E. S. (2026, August). Chronological thinking in full-duplex spoken dialogue language models. In Proceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue pp\. 473-485.
[2] Wu, D., Zhang, T., Li, Y., Liu, H., Chen, C., Chng, E. S., & Bengio, Y. . The silent thought: Modeling internal cognition in full-duplex spoken dialogue models via latent reasoning. arXiv preprint arXiv:2603.17837.
Author Response and Additional Experimental Results for Reviewer 5fML
We thank the reviewer for the thoughtful and constructive feedback, and for recognizing the contributions of FDGym and the effectiveness of our approach. We appreciate the detailed technical questions and suggestions, which help us clarify the motivation, implementation, and evaluation of our method. We address each concern below.
W1: Positioning and scope.
We agree that character consistency is not unique to full-duplex models. Our central methodological contribution is FDGym, an interactive RL framework for full-duplex dialogue, with character consistency as the application studied in this paper. Importantly, FDGym explicitly addresses full-duplex-specific interaction behaviors rather than optimizing character consistency alone. We provide two complementary experimental results.
(1) Timing rewards are essential for full-duplex interaction. FDGym incorporates two timing rewards: r_resp for response initiation and r_yield for yielding under user interruption. Removing both rewards leaves character performance nearly unchanged (SpeechRole-Eval Avg: 0.784 vs. 0.785), but substantially degrades full-duplex interaction:
Without timing rewards, the model produces no responses on the Candor subset, and its interruption TOR drops from 0.990 to 0.440. These results demonstrate that content-based rewards alone are insufficient to maintain full-duplex interaction behavior during RL, and that explicit timing optimization is critical.
(2) Response-dependent interaction improves multi-turn RL. We compare FDGym against two fixed-user RL baselines. In particular, Fixed-all uses the same initialization, rewards, GRPO settings, datasets and optimization over all assistant turns, but removes the interactive closed loop by keeping subsequent user utterances fixed. Fixed-last additionally uses ground-truth assistant prefixes and optimizes only the final response.
Both fixed-user baselines fall below SFT, while FDGym improves all four character-related metrics. In particular, the comparison with Fixed-all supports the importance of response-dependent user continuations for effective multi-turn RL.
Together, these results establish two important aspects of FDGym: explicit optimization of full-duplex interaction timing and response-dependent closed-loop training.
W2: Backbone motivation and comparison with TurnGuide.
Thank you for pointing us to TurnGuide. We clarify our design motivation and the differences below.
Motivation. Our architectural design follows prior work rather than constituting a new contribution. Specifically, we adopt the dual-stream formulation of Chronological Thinking [1] and the decoupled speech generation design of FLAIR [2]. Neither work has published codes or weights. We implement this architecture on the open-source GLM-4-Voice backbone, retaining its pretrained speech tokenizer and LLM while adapting the output stream to predict text and turn-control tokens.
Although GLM-4-Voice originally supports speech-to-speech generation, we deliberately decouple language modeling from acoustic generation. This allows the LLM to focus on response content and temporal interaction control, which stays close to text language modeling, while a separate module handles speech synthesis. The resulting system remains speech-to-speech, but the LLM no longer needs to generate acoustic tokens directly.
Comparison with TurnGuide. Both methods adapt GLM-4-Voice to full-duplex interaction, but differ in their modeling approaches:
PAD/BOS/EOS)We do not claim this architectural choice as a novel contribution or establish its superiority over TurnGuide. Our listed contributions (Section 1) do not include these architectural designs. Rather, it provides the backbone for our character-conditioned training pipeline and FDGym. We have clarified this motivation and add the TurnGuide comparison in the revised paper.