- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CSDLM, a continuous-latent spoken dialogue language model that unifies two-channel dialogue modeling and speech generation through interleaved text-audio sequences and autoregressive flow matching. The framework supports dialogue continuation, dialogue generation, contextual TTS, and streaming TTS without task-specific fine-tuning, achieving competitive performance across multiple tasks.
Critical strengths
-
The paper addresses an interesting and practically relevant problem of unified spoken dialogue modeling and speech generation.
-
The framework supports speech continuation and generation tasks within a single model, demonstrating promising versatility.
-
The experimental results are encouraging, particularly in dialogue coherence, speaker similarity, and streaming latency.
Critical weaknesses
-
Limited methodological novelty. Several key components have been explored in prior work. CoVoMix [1] investigates multi-talker conversational speech generation, while Alignment-Free Text-Audiobox [2] explores continuous-latent flow matching for full-duplex dialogue synthesis. Moreover, Delayed Streams Modeling [3] has investigated streaming generation through time-aligned multimodal streams. Together with existing continuous speech generation models such as VibeVoice, these studies raise questions about the distinct methodological contribution of CSDLM. The authors should better differentiate their approach from these related works.
-
Insufficient ablation studies. It remains unclear whether the performance improvements originate from continuous audio latents, text-audio interleaving, or two-channel sequence modeling. Systematic ablations are needed to establish the individual contributions and necessity of these components.
-
Insufficient validation of overlap controllability. The proposed BOS thresholding mechanism primarily controls overall speaking activity rather than the precise timing of overlapping speech. Furthermore, overlaps are explicitly restricted during dialogue generation evaluation. Consequently, the model's ability to generate natural and temporally controllable overlapping speech remains insufficiently demonstrated.
[1] Zhang, L., et al. CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations. NeurIPS, 2024.
[2] Chen, S., et al. Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis. arXiv preprint arXiv:2609.03992, 2026.
[3] Zeghidour, N., et al. Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling. arXiv preprint arXiv:2509.08753, 2025.
Questions for authors
-
How is the BOS threshold selected during dialogue continuation? Given its substantial influence on turn-taking statistics, is there a principled selection criterion or an adaptive strategy based on conversational context?
-
Dialogue continuation involves unconstrained generation of both text and speech, whereas dialogue generation conditions on predefined text. What evidence demonstrates that the learned dialogue modeling capabilities improve text-conditioned dialogue generation?
-
Can CSDLM explicitly control the temporal positions of overlapping speech or backchannels? How does this capability compare with existing multi-talker dialogue synthesis approaches?
6 comments
Response to Reviewer Z2ZT (2/3)
Response to the third weakness and the first question
We clarify that BOS thresholding is an optional inference-time control for adjusting speaking activity. By default, CSDLM generates text and speech autoregressively and emits BOS according to its learned token distribution, without applying an additional BOS threshold.
The fixed thresholds reported in the paper demonstrate how the frequency and amount of speech change as the threshold varies. We do not use a separate context-adaptive threshold policy.
CSDLM implicitly learns context-dependent BOS predictions during autoregressive training, which determine when each speaker begins speaking. For explicit control over utterance onset, BOS can be forced at a desired audio-frame position, as described in our response to the third question.
To evaluate the naturalness of CSDLM’s turn-taking behavior, we conducted a human listening study. The current results are based on ratings from nine participants. We will continue recruiting participants throughout the rebuttal period and update the results as more ratings are collected. All CSDLM continuations used in our additional human listening study were generated without BOS thresholding. The turn-taking naturalness criterion includes the appropriateness of speaker changes, pauses, overlaps, and backchannels. CSDLM received a mean turn-taking naturalness score of 4.21 on a 1–5 scale, compared with 3.08 for dGSLM and 4.07 for ground truth (Table 3). These results support CSDLM’s ability to generate natural turn-taking without manual BOS-threshold control. We have added the evaluation protocol and results in blue to the revised paper (Section 3.2 and Appendix B.1).
| Evaluation criterion | Ground Truth | dGSLM | CSDLM |
|---|---|---|---|
| Semantic coherence | 4.10 | 1.99 | 4.21 |
| Turn-taking naturalness | 4.07 | 3.08 | 4.21 |
| Speaker similarity | 4.51 | 2.07 | 4.40 |
| Emotional coherence | 4.28 | 2.56 | 4.21 |
| Audio quality | 3.85 | 1.69 | 4.04 |
| Overall conversational naturalness | 3.96 | 1.93 | 4.06 |
The non-overlap constraint in our dialogue-generation evaluation was imposed to match the turn-based structure of the benchmark. Our updated dialogue-generation demos include cases in which one speaker’s entire utterance occurs during the other speaker’s ongoing speech.
Response to Reviewer Z2ZT (3/3)
Response to the second question
CSDLM is trained with a dialogue continuation objective, learning to continue two-channel conversations through autoregressive prediction of text tokens and continuous audio latents. We use the same trained checkpoint for text-conditioned speech generation, without task-specific fine-tuning, changing only the inference procedure. The model therefore draws on the dialogue modeling capabilities acquired during training to generate speech and coordinate both speakers under the supplied text.
Dialogue continuation requires the model to use the preceding conversation to generate contextually appropriate text and speech. Our contextual TTS experiment provides quantitative evidence that this ability to use conversational context also benefits speech generation when the target text is predefined. We compare the same model with and without preceding conversational context while keeping the target text and timbre conditioning unchanged. Adding context improves EmoSIM from 0.67 to 0.72, reduces WER from 14.84% to 9.09%, and increases SIM from 0.40 to 0.47. These results indicate that CSDLM uses conversational context to better model emotional expression and speaker characteristics while preserving text fidelity.
We also invite the reviewer to listen to the Dialogue Generation examples on our updated demo page, which include overlapping speech and backchannels generated from predefined dialogue scripts. These examples illustrate how the dialogue modeling capabilities learned through our unified two-channel sequence modeling support coordinated speech generation across both channels when the dialogue text is predefined.
Response to the third question
Yes. CSDLM can explicitly control utterance onset by forcing a BOS token on the target channel at a desired audio-frame position. Because continuous audio latents have a fixed frame rate and the two channels are time-aligned, the audio-frame index determines the corresponding time in the conversation.
Our updated dialogue-generation demos include not only brief overlaps but also cases in which one speaker’s entire utterance occurs during the other speaker’s ongoing speech. We invite the reviewer to listen to these examples in the Dialogue Generation section of our updated demo page.
Existing multi-talker dialogue synthesis approaches often have limited support for overlapping speech. VibeVoice [4] does not explicitly support overlapping speech, while ZipVoice-Dialog-Stereo [6] introduces a speaker-exclusive loss to penalize overlap after observing that simultaneous speech typically degrades the quality of one channel. CoVoMix2 [7] supports overlap and timing control through predefined utterance intervals, with durations computed from syllable counts and a predefined speaking rate. However, we could not find a publicly released model implementation, and its public overlap demos consist of short, two-turn exchanges, each with a single interruption. These examples do not demonstrate more varied overlapping interactions such as backchannels. It also generates mixed single-channel speech. On the other hand, CSDLM explicitly models each speaker’s audio in a separate, time-aligned channel through the proposed interleaved sequence, allowing it to learn cross-speaker interactions, including overlap. Training this sequence with a next-token prediction objective enables CSDLM to learn dialogue semantics and turn-taking while supporting explicit onset control at inference time without additional fine-tuning.
References
[4] Peng et al., VibeVoice Technical Report, 2025.
[6] Zhu et al., ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matching, 2025.
[7] Zhang et al., CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching, 2025.
Response to Reviewer Z2ZT (1/3)
We thank the reviewer for their constructive feedback. We address the weaknesses and questions below.
Response to the first weakness
We clarify below how CSDLM differs from these related studies. Our core contribution is a speech generation foundation model that learns semantic content, turn-taking, and acoustic context from two-channel spoken conversations in continuous latent space. Our methodological contribution is a sequence formulation that interleaves text tokens and continuous audio latents within each speaker channel and serializes time-aligned chunks from both channels into a single autoregressive sequence. This formulation enables CSDLM to learn foundational dialogue modeling capabilities while allowing fine-grained control over the timing of overlaps and backchannels at inference time. It also enables the same checkpoint to perform multiple speech generation tasks without task-specific fine-tuning.
We also investigate four voice-prompt conditioning designs for autoregressive flow-matching speech generation. Conditioning both the LM prefix and the flow-matching head on speaker embeddings yields the highest speaker similarity among the four evaluated designs.
Among the models compared above, CSDLM is the only model that supports dialogue continuation. This task requires the model to determine not only how and when to speak, but also what to say next based on the preceding conversational context. CSDLM jointly models dialogue semantics, turn-taking, and acoustics to generate both speakers’ text and speech without a predefined continuation transcript.
CSDLM is also the only model among those compared above that combines two-channel speech modeling with support for streaming text input. This combination facilitates its use in interactive systems with distinct user and agent roles.
We also note that Alignment-Free Text-Audiobox [2] was first posted on arXiv approximately three weeks before the submission deadline. The ICLR reviewer guidelines [5] state that comparisons with contemporaneous or unpublished arXiv work are not required and that “the lack of such comparisons cannot be a basis for rejection.” We nevertheless discuss this related work here to clarify the methodological differences.
Response to the second weakness
We agree that ablation studies would help clarify the contributions of individual design choices.
Our design relies on the proposed interleaved sequence to model dialogue semantics and turn-taking across the two speaker channels. Our motivation for using continuous audio latents is to capture fine-grained acoustic information within the autoregressive backbone, including speaker timbre and context-dependent emotional expression.
To address the reviewer’s suggestion, we are currently conducting experiments with discrete-representation variants of CSDLM using the same backbone, pretrained initialization, and training data. We will update our response and the paper with results as they become available during the discussion period.
References
[1] Zhang et al., CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations, 2024.
[2] Chen et al., Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis, 2026.
[3] Zeghidour et al., Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling, 2025.
[4] Peng et al., VibeVoice Technical Report, 2025.
[5] ICLR 2027 Reviewer Guidelines.