acceptodds
Under review as a conference paper at ICLR 2027

Continuous Spoken Dialogue Language Models

Abstract

Spoken dialogue language models (SDLMs) can support spoken interaction by modeling turn-taking behavior and semantic content. However, most existing SDLMs operate on semantic speech tokens, which may limit the modeling of fine-grained acoustic information. Furthermore, it is unclear whether their dialogue understanding benefits diverse speech generation tasks. In this work, we introduce Continuous Spoken Dialogue Language Model (CSDLM), a foundation model that models two-channel spoken dialogue in a continuous latent space. By modeling time-aligned channels through a unified interleaved text-speech stream, CSDLM supports diverse tasks, including both single-channel and two-channel speech generation, without task-specific fine-tuning. We also investigate effective voice-prompt conditioning designs with classifier-free guidance to improve speaker similarity in autoregressive flow-matching speech generation settings. We evaluate CSDLM on a suite of tasks including dialogue continuation, dialogue generation, contextual text-to-speech (TTS), and streaming TTS. Compared with other spoken dialogue models like dGSLM, CSDLM enables fine-grained control over speaking activity while improving the semantic coherence of the generated dialogue by almost 2x. Despite being a streaming model, it matches the lexical and timbre fidelity (in terms of word error rate and speaker similarity, respectively) of non-streaming baselines like VibeVoice and ZipVoice-Dialog, while generating channel-separated dialogue from a reference text. Audio samples are available on the project page: https://anonymous.4open.science/w/temp-3B2C/

Primary area:
generative models
Keywords:
Spoken Dialogue Language Models, Continuous Modeling, Flow Matching, Streaming Speech Generation, Text-to-Speech

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Z2ZTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CSDLM, a continuous-latent spoken dialogue language model that unifies two-channel dialogue modeling and speech generation through interleaved text-audio sequences and autoregressive flow matching. The framework supports dialogue continuation, dialogue generation, contextual TTS, and streaming TTS without task-specific fine-tuning, achieving competitive performance across multiple tasks.

Critical strengths
  1. The paper addresses an interesting and practically relevant problem of unified spoken dialogue modeling and speech generation.

  2. The framework supports speech continuation and generation tasks within a single model, demonstrating promising versatility.

  3. The experimental results are encouraging, particularly in dialogue coherence, speaker similarity, and streaming latency.

Critical weaknesses
  1. Limited methodological novelty. Several key components have been explored in prior work. CoVoMix [1] investigates multi-talker conversational speech generation, while Alignment-Free Text-Audiobox [2] explores continuous-latent flow matching for full-duplex dialogue synthesis. Moreover, Delayed Streams Modeling [3] has investigated streaming generation through time-aligned multimodal streams. Together with existing continuous speech generation models such as VibeVoice, these studies raise questions about the distinct methodological contribution of CSDLM. The authors should better differentiate their approach from these related works.

  2. Insufficient ablation studies. It remains unclear whether the performance improvements originate from continuous audio latents, text-audio interleaving, or two-channel sequence modeling. Systematic ablations are needed to establish the individual contributions and necessity of these components.

  3. Insufficient validation of overlap controllability. The proposed BOS thresholding mechanism primarily controls overall speaking activity rather than the precise timing of overlapping speech. Furthermore, overlaps are explicitly restricted during dialogue generation evaluation. Consequently, the model's ability to generate natural and temporally controllable overlapping speech remains insufficiently demonstrated.

[1] Zhang, L., et al. CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations. NeurIPS, 2024.

[2] Chen, S., et al. Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis. arXiv preprint arXiv:2609.03992, 2026.

[3] Zeghidour, N., et al. Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling. arXiv preprint arXiv:2509.08753, 2025.

Questions for authors
  1. How is the BOS threshold selected during dialogue continuation? Given its substantial influence on turn-taking statistics, is there a principled selection criterion or an adaptive strategy based on conversational context?

  2. Dialogue continuation involves unconstrained generation of both text and speech, whereas dialogue generation conditions on predefined text. What evidence demonstrates that the learned dialogue modeling capabilities improve text-conditioned dialogue generation?

  3. Can CSDLM explicitly control the temporal positions of overlapping speech or backchannels? How does this capability compare with existing multi-talker dialogue synthesis approaches?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Z2ZT (1/3)

We thank the reviewer for their constructive feedback. We address the weaknesses and questions below.

Response to the first weakness

We clarify below how CSDLM differs from these related studies. Our core contribution is a speech generation foundation model that learns semantic content, turn-taking, and acoustic context from two-channel spoken conversations in continuous latent space. Our methodological contribution is a sequence formulation that interleaves text tokens and continuous audio latents within each speaker channel and serializes time-aligned chunks from both channels into a single autoregressive sequence. This formulation enables CSDLM to learn foundational dialogue modeling capabilities while allowing fine-grained control over the timing of overlaps and backchannels at inference time. It also enables the same checkpoint to perform multiple speech generation tasks without task-specific fine-tuning.

We also investigate four voice-prompt conditioning designs for autoregressive flow-matching speech generation. Conditioning both the LM prefix and the flow-matching head on speaker embeddings yields the highest speaker similarity among the four evaluated designs.

ModelDialogue continuationStreaming text inputOutput channels
CoVoMix [1]××1ch / 2ch (separate variants)
Text-Audiobox [2]××1ch / 2ch (separate variants)
DSM-TTS [3]×○1ch
VibeVoice [4]××1ch
CSDLM○○1ch and 2ch (same checkpoint)

Among the models compared above, CSDLM is the only model that supports dialogue continuation. This task requires the model to determine not only how and when to speak, but also what to say next based on the preceding conversational context. CSDLM jointly models dialogue semantics, turn-taking, and acoustics to generate both speakers’ text and speech without a predefined continuation transcript.

CSDLM is also the only model among those compared above that combines two-channel speech modeling with support for streaming text input. This combination facilitates its use in interactive systems with distinct user and agent roles.

We also note that Alignment-Free Text-Audiobox [2] was first posted on arXiv approximately three weeks before the submission deadline. The ICLR reviewer guidelines [5] state that comparisons with contemporaneous or unpublished arXiv work are not required and that “the lack of such comparisons cannot be a basis for rejection.” We nevertheless discuss this related work here to clarify the methodological differences.

Response to the second weakness

We agree that ablation studies would help clarify the contributions of individual design choices.

Our design relies on the proposed interleaved sequence to model dialogue semantics and turn-taking across the two speaker channels. Our motivation for using continuous audio latents is to capture fine-grained acoustic information within the autoregressive backbone, including speaker timbre and context-dependent emotional expression.

To address the reviewer’s suggestion, we are currently conducting experiments with discrete-representation variants of CSDLM using the same backbone, pretrained initialization, and training data. We will update our response and the paper with results as they become available during the discussion period.

References

[1] Zhang et al., CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations, 2024.

[2] Chen et al., Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis, 2026.

[3] Zeghidour et al., Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling, 2025.

[4] Peng et al., VibeVoice Technical Report, 2025.

[5] ICLR 2027 Reviewer Guidelines.

AuthorsOpenReview ↗

Response to Reviewer Z2ZT (2/3)

Response to the third weakness and the first question

We clarify that BOS thresholding is an optional inference-time control for adjusting speaking activity. By default, CSDLM generates text and speech autoregressively and emits BOS according to its learned token distribution, without applying an additional BOS threshold.

The fixed thresholds reported in the paper demonstrate how the frequency and amount of speech change as the threshold varies. We do not use a separate context-adaptive threshold policy.

CSDLM implicitly learns context-dependent BOS predictions during autoregressive training, which determine when each speaker begins speaking. For explicit control over utterance onset, BOS can be forced at a desired audio-frame position, as described in our response to the third question.

To evaluate the naturalness of CSDLM’s turn-taking behavior, we conducted a human listening study. The current results are based on ratings from nine participants. We will continue recruiting participants throughout the rebuttal period and update the results as more ratings are collected. All CSDLM continuations used in our additional human listening study were generated without BOS thresholding. The turn-taking naturalness criterion includes the appropriateness of speaker changes, pauses, overlaps, and backchannels. CSDLM received a mean turn-taking naturalness score of 4.21 on a 1–5 scale, compared with 3.08 for dGSLM and 4.07 for ground truth (Table 3). These results support CSDLM’s ability to generate natural turn-taking without manual BOS-threshold control. We have added the evaluation protocol and results in blue to the revised paper (Section 3.2 and Appendix B.1).

Evaluation criterionGround TruthdGSLMCSDLM
Semantic coherence4.101.994.21
Turn-taking naturalness4.073.084.21
Speaker similarity4.512.074.40
Emotional coherence4.282.564.21
Audio quality3.851.694.04
Overall conversational naturalness3.961.934.06

The non-overlap constraint in our dialogue-generation evaluation was imposed to match the turn-based structure of the benchmark. Our updated dialogue-generation demos include cases in which one speaker’s entire utterance occurs during the other speaker’s ongoing speech.

AuthorsOpenReview ↗

Response to Reviewer Z2ZT (3/3)

Response to the second question

CSDLM is trained with a dialogue continuation objective, learning to continue two-channel conversations through autoregressive prediction of text tokens and continuous audio latents. We use the same trained checkpoint for text-conditioned speech generation, without task-specific fine-tuning, changing only the inference procedure. The model therefore draws on the dialogue modeling capabilities acquired during training to generate speech and coordinate both speakers under the supplied text.

Dialogue continuation requires the model to use the preceding conversation to generate contextually appropriate text and speech. Our contextual TTS experiment provides quantitative evidence that this ability to use conversational context also benefits speech generation when the target text is predefined. We compare the same model with and without preceding conversational context while keeping the target text and timbre conditioning unchanged. Adding context improves EmoSIM from 0.67 to 0.72, reduces WER from 14.84% to 9.09%, and increases SIM from 0.40 to 0.47. These results indicate that CSDLM uses conversational context to better model emotional expression and speaker characteristics while preserving text fidelity.

We also invite the reviewer to listen to the Dialogue Generation examples on our updated demo page, which include overlapping speech and backchannels generated from predefined dialogue scripts. These examples illustrate how the dialogue modeling capabilities learned through our unified two-channel sequence modeling support coordinated speech generation across both channels when the dialogue text is predefined.

Response to the third question

Yes. CSDLM can explicitly control utterance onset by forcing a BOS token on the target channel at a desired audio-frame position. Because continuous audio latents have a fixed frame rate and the two channels are time-aligned, the audio-frame index determines the corresponding time in the conversation.

Our updated dialogue-generation demos include not only brief overlaps but also cases in which one speaker’s entire utterance occurs during the other speaker’s ongoing speech. We invite the reviewer to listen to these examples in the Dialogue Generation section of our updated demo page.

Existing multi-talker dialogue synthesis approaches often have limited support for overlapping speech. VibeVoice [4] does not explicitly support overlapping speech, while ZipVoice-Dialog-Stereo [6] introduces a speaker-exclusive loss to penalize overlap after observing that simultaneous speech typically degrades the quality of one channel. CoVoMix2 [7] supports overlap and timing control through predefined utterance intervals, with durations computed from syllable counts and a predefined speaking rate. However, we could not find a publicly released model implementation, and its public overlap demos consist of short, two-turn exchanges, each with a single interruption. These examples do not demonstrate more varied overlapping interactions such as backchannels. It also generates mixed single-channel speech. On the other hand, CSDLM explicitly models each speaker’s audio in a separate, time-aligned channel through the proposed interleaved sequence, allowing it to learn cross-speaker interactions, including overlap. Training this sequence with a next-token prediction objective enables CSDLM to learn dialogue semantics and turn-taking while supporting explicit onset control at inference time without additional fine-tuning.

References

[4] Peng et al., VibeVoice Technical Report, 2025.

[6] Zhu et al., ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matching, 2025.

[7] Zhang et al., CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching, 2025.

Reviewer_ewRCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CSDLM, a foundation model for two-channel spoken dialogue built on autoregressive flow matching over continuous VAE latents (the VibeVoice VAE, 7.5 Hz). Unlike prior SDLMs that operate on discrete semantic tokens or CALM's multi-stream summed embeddings, CSDLM serializes text and audio from both speakers into a single interleaved sequence. Within each inter-pausal unit (IPU) it alternates ntn_t text tokens with nan_a audio latents (nt:na=5:6n_t:n_a = 5:6), delimited by channel-specific bos/eot/eos tokens. The two channels are interleaved as time-aligned chunks in a fixed order, with one latent per chunk (Sec. 2.2). The backbone and LM head are initialized from Qwen3.5-2B-Base, and a 50M-parameter MLP flow-matching head generates latents at audio positions. Four CFG-based voice-prompt conditioning designs are compared (Table 1), with dual-path speaker-embedding conditioning giving the best WER/SIM balance.

Critical strengths

A clean, general sequence design that unifies dialogue modeling and speech generation in one checkpoint. Interleaving per-IPU text/latent segments and then serializing time-aligned chunks across channels lets one autoregressive model jointly capture semantics, turn-taking, and acoustics. Without task-specific fine-tuning, the same checkpoint then supports dialogue continuation, script-to-dialogue generation, contextual TTS, and streaming TTS.

Critical weaknesses
  1. The paper motivates CSDLM by arguing that discrete semantic tokens "may limit the modeling of fine-grained acoustic information" . Yet no discrete-token variant of the same architecture (same backbone, data, and interleaving) is trained. The only SDLM comparison is dGSLM , which differs in almost every respect: no text stream (CSDLM predicts aligned text before speech, which Sec. 2.2 itself credits with better semantic modeling), no pretrained LLM initialization (CSDLM uses Qwen3.5-2B), and different training data and scale. The "almost 2× coherence" claim therefore cannot be attributed to continuous latents and is most plausibly explained by text interleaving and LLM initialization. Choosing dGSLM is justified by public availability, but this still leaves a single, pre-LLM baseline.
  2. Although termed a Spoken Dialogue Language Model, CSDLM is primarily a speech generation foundation model rather than a unified speech understanding and generation model, making it less suitable for speech understanding tasks.
  3. There is no MOS, CMOS, or naturalness/turn-taking preference test anywhere, despite claims about acoustic fidelity and expressive conversational speech.
  4. Core design choices are not ablated: the 5:6 interleaving ratio, the chunk size of 1, the 1:14 monologue:dialogue sampling ratio, λART=0.3\lambda_{ART}=0.3, and the necessity of two stages.
Questions for authors

Can you isolate the contribution of continuous latents from that of text interleaving and Qwen initialization? For example, a discrete-token CSDLM with the same backbone and data, or a CSDLM trained without the text stream

2 comments
AuthorsOpenReview ↗

Response to Reviewer ewRC

We thank the reviewer for their constructive feedback. We address the weaknesses and questions below.

Response to the first weakness and the question

We agree that a comparison under matched training conditions would help isolate the contribution of continuous representations.

Our design relies on the proposed interleaved sequence to model dialogue semantics and turn-taking across the two speaker channels. Our motivation for using continuous audio latents is to capture fine-grained acoustic information within the autoregressive backbone, including speaker timbre and context-dependent emotional expression.

The “almost 2× coherence” improvement reported in the paper refers to the comparison between the full CSDLM system and dGSLM. We do not attribute this improvement to continuous audio latents alone.

To address the reviewer’s suggestion, we are currently conducting experiments with discrete-representation variants of CSDLM using the same backbone, pretrained initialization, and training data. We will update our response and the paper with results as they become available during the discussion period.

Response to the second weakness

Our goal is to model the semantic content, turn-taking behavior, and acoustic attributes of two-channel spoken dialogue and to use these learned conversational capabilities across multiple speech generation tasks. We use “Spoken Dialogue Language Model” to describe CSDLM’s autoregressive modeling of spoken conversations, which involves generating both text and speech. Just as coherent text continuation requires a language model to understand linguistic context, coherent spoken dialogue continuation requires a spoken dialogue language model to understand conversational context. CSDLM must determine what to say next, as well as how and when to speak. This goes beyond synthesizing speech from a predefined transcript. We leave extending CSDLM to a broader range of speech-understanding tasks to future work.

Response to the third weakness

Following the reviewer's suggestion, we conducted a human listening study to evaluate the acoustic quality, emotional coherence, turn-taking naturalness, and overall naturalness of generated conversations. The current results are based on ratings from nine participants. We will continue recruiting participants throughout the rebuttal period and update the results as more ratings are collected. We have added the evaluation protocol and results in blue to the revised paper (Section 3.2 and Appendix B.1).

After listening to each prompt, participants independently rated continuations from CSDLM, dGSLM, and ground truth on a 1~5 scale for semantic coherence, turn-taking naturalness, speaker similarity, emotional coherence, audio quality, and overall conversational naturalness. Continuations were presented without revealing their source identities. The order of evaluation items and the order of continuations within each item were randomized for each participant. They assessed turn-taking in the context of the preceding conversation, considering speaker changes, pauses, overlaps, and backchannels, and judged whether the emotional expression fit the conversational context.

Evaluation criterionGround TruthdGSLMCSDLM
Semantic coherence4.101.994.21
Turn-taking naturalness4.073.084.21
Speaker similarity4.512.074.40
Emotional coherence4.282.564.21
Audio quality3.851.694.04
Overall conversational naturalness3.961.934.06

CSDLM received substantially higher mean ratings than dGSLM across all six criteria. Its mean ratings were also close to those of ground truth, with a maximum shortfall of 0.11 points. These ratings suggest that CSDLM generates dialogue with audio quality, emotional coherence, and turn-taking naturalness close to those of human conversations.

Response to the fourth weakness

The text-audio interleaving ratio balances text availability and streaming latency. A higher proportion of audio can require the model to generate speech before receiving the corresponding text, whereas a higher proportion of text can delay speech generation by requiring more text input first. In our training data, approximately 90% of utterances contained more than 1.2 audio latents per text token. Based on this distribution, we selected a text–audio ratio of 5:6.

Given the time constraints of the rebuttal period, we have not yet completed systematic ablations of the remaining design choices. We will share any additional ablation results that become available during the discussion period.

AuthorsOpenReview ↗

Updated ablation results and listening study

We have completed the matched-condition ablations and incorporated the new results and updated listening study into the revised manuscript.

Representation and text-stream ablations (W1/Q1)

To evaluate the role of continuous audio representations, we compare CSDLM with a discrete-token variant, using Mimi RVQ [1] as the discrete representation. We replace the flow-matching head with a depth transformer [1] to predict audio tokens across the RVQ codebooks. To assess the text stream’s contribution to learning dialogue semantics and turn-taking, we also evaluate a variant that removes it while retaining continuous audio latents. All variants use the same backbone, Qwen3.5-2B-Base initialization, and training data, with 20K steps in each training stage.

Dialogue continuation

ModelHead params.GPT-Score ↑PPLIPUPauseGapOverlap
Ground Truth—3.828.349.3 (49.2%)4.3 (5.9%)5.6 (5.0%)7.8 (9.6%)
CSDLM47.6M2.636.579.9 (38.8%)6.0 (9.7%)5.4 (6.5%)7.7 (7.3%)
w/ Mimi RVQ645.9M2.307.098.8 (37.0%)2.3 (3.5%)3.9 (3.6%)10.6 (15.4%)
w/o text47.6M1.09266.231.5 (2.7%)1.6 (20.1%)0.6 (10.0%)0.1 (0.1%)

Text-conditioned generation

ModelHead params.Dialogue generation WER (%) ↓Dialogue generation SIM ↑Contextual TTS WER (%) ↓Contextual TTS SIM ↑Contextual TTS EmoSIM ↑TTS WER (%) ↓TTS SIM ↑
CSDLM47.6M10.610.5013.420.400.696.330.53
w/ Mimi RVQ645.9M14.100.4517.940.310.678.000.44

Compared with the Mimi RVQ variant, CSDLM achieves a higher GPT-Score on dialogue continuation, lower WER and higher SIM across all three text-conditioned generation tasks, and higher EmoSIM on contextual TTS, while using a substantially smaller audio prediction head. The gains in SIM and EmoSIM suggest that continuous-latent modeling helps capture fine-grained acoustic attributes, including speaker timbre and context-dependent emotional expression.

Removing the text stream substantially reduces GPT-Score and increases PPL, accompanied by a sharp reduction in speaking activity. These results support the text stream’s contribution to learning dialogue semantics and turn-taking under the evaluated training conditions. Our semantic findings are consistent with prior work showing that an internal text stream improves semantic coherence and accelerates convergence in speech language models [2].

The dialogue continuation ablation results are now included in Table 2 and discussed in Section 3.6. Results on the text-conditioned generation tasks, full experimental details, and further analysis are provided in Appendix C.

Updated listening study (W3)

We have expanded the listening study from 9 to 12 participants using the same evaluation protocol. The table below reports the updated means and 95% confidence intervals.

Evaluation criterionGround TruthdGSLMCSDLM
Semantic coherence4.05 ± 0.301.91 ± 0.684.17 ± 0.34
Turn-taking naturalness4.15 ± 0.383.04 ± 0.524.26 ± 0.30
Speaker similarity4.56 ± 0.182.02 ± 0.694.44 ± 0.24
Emotional coherence4.31 ± 0.272.48 ± 0.624.22 ± 0.30
Audio quality3.93 ± 0.261.79 ± 0.494.16 ± 0.31
Overall conversational naturalness4.01 ± 0.241.91 ± 0.584.09 ± 0.25

The overall trend remains unchanged: CSDLM receives substantially higher mean ratings than dGSLM across all six criteria, with ratings close to those of ground truth. The updated results are included in Table 3, and the confidence-interval calculation is described in Appendix B.1. We have also added all eight listening-study examples and the aggregate ratings to the Listening study samples subsection of our demo page.

References

[1] Défossez et al., Moshi: a speech-text foundation model for real-time dialogue, 2024.

[2] Chou et al., Quantifying the Generation Modality Gap in Speech-Text Language Models, 2026.

Reviewer_A6sDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CSDLM, a model for two-channel spoken dialogue in a continuous audio latent space. It interleaves text and audio from both channels into a single temporally aligned sequence and models it using an autoregressive Transformer and a flow-matching head. A single model supports dialogue continuation, dialogue generation, contextual TTS, and streaming TTS, achieving competitive performance across these tasks.

Critical strengths

S1. Supporting single-channel and two-channel speech generation, text-conditioned generation, and dialogue continuation within one model is useful. Extending dialogue modeling capabilities to diverse speech generation tasks is also a meaningful direction.

S2. The figures clearly illustrate the sequence construction and model architecture, making the proposed framework easier to understand.

Critical weaknesses

W1. Although the overall framework is effective, the contributions of its core design choices remain unclear. Compared with dGSLM, CSDLM differs not only in continuous versus discrete representations but also in text conditioning, the backbone, and training data. It is therefore difficult to isolate the contribution of continuous representations to improved dialogue quality. A comparison under more closely matched training conditions would strengthen the evidence.

W2. The reported evaluation relies on automatic metrics and would benefit from human listening assessments. WER, speaker similarity, emotion similarity, and dialogue statistics provide useful evidence, but do not directly establish whether listeners perceive the generated conversations as natural, contextually appropriate, and emotionally coherent. In particular, controlling overlap frequency does not establish whether the overlapping speech or backchannels are appropriate in context.

Questions for authors

Q1. Could the authors compare continuous and discrete representations using the same backbone and training data where feasible? This would help distinguish the contribution of continuous representations from other design differences.

Q2. Could the authors complement the automatic metrics with a human listening study comparing CSDLM and the baselines? It would be helpful to assess whether the conversations sound natural, convey suitable emotions, and use overlaps and backchannels appropriately.

4 comments
AuthorsOpenReview ↗

Response to Reviewer A6sD

We thank the reviewer for their constructive feedback. We address the weaknesses and questions below.

Response to W1 and Q1

We agree that a comparison under matched training conditions would help isolate the contribution of continuous representations.

Our design relies on the proposed interleaved sequence to model dialogue semantics and turn-taking across the two speaker channels. Our motivation for using continuous audio latents is to capture fine-grained acoustic information within the autoregressive backbone, including speaker timbre and context-dependent emotional expression.

To address the reviewer’s suggestion, we are currently conducting experiments with discrete-representation variants of CSDLM using the same backbone and training data. We will update our response and the paper with results as they become available during the discussion period.

Response to W2 and Q2

Following the reviewer's suggestion, we conducted a human listening study to evaluate the naturalness and contextual appropriateness of generated conversations. The current results are based on ratings from nine participants. We will continue recruiting participants throughout the rebuttal period and update the results as more ratings are collected. We have added the evaluation protocol and results in blue to the revised paper (Section 3.2 and Appendix B.1).

After listening to each prompt, participants independently rated continuations from CSDLM, dGSLM, and ground truth on a 1~5 scale for semantic coherence, turn-taking naturalness, speaker similarity, emotional coherence, audio quality, and overall conversational naturalness. Continuations were presented without revealing their source identities. The order of evaluation items and the order of continuations within each item were randomized for each participant. They assessed turn-taking in the context of the preceding conversation, considering speaker changes, pauses, overlaps, and backchannels, and judged whether the emotional expression fit the conversational context.

Evaluation criterionGround TruthdGSLMCSDLM
Semantic coherence4.101.994.21
Turn-taking naturalness4.073.084.21
Speaker similarity4.512.074.40
Emotional coherence4.282.564.21
Audio quality3.851.694.04
Overall conversational naturalness3.961.934.06

CSDLM received substantially higher mean ratings than dGSLM across all six criteria. Its mean ratings were also close to those of ground truth, with a maximum shortfall of 0.11 points. These ratings suggest that CSDLM generates dialogue with turn-taking naturalness, semantic coherence, and emotional coherence close to those of human conversations.

Thank you for your detailed response and for conducting the human listening study. I look forward to the results of the matched-condition experiments addressing W1. Could you also provide 95% confidence intervals for the human evaluation scores? Additionally, how should we interpret CSDLM’s higher mean ratings than ground truth on several criteria?

Thank you for these follow-up questions. We report the scores with 95% confidence intervals below. For each criterion and model, we first averaged each participant’s ratings across the eight dialogue items, then computed a Student-t confidence interval across participants. These intervals reflect participant variability for the evaluated dialogue items.

Values are means ± 95% confidence-interval half-widths.

Evaluation criterionGround TruthdGSLMCSDLM
Semantic coherence4.10 ± 0.421.99 ± 0.934.21 ± 0.42
Turn-taking naturalness4.07 ± 0.463.08 ± 0.704.21 ± 0.39
Speaker similarity4.51 ± 0.222.07 ± 0.944.40 ± 0.30
Emotional coherence4.28 ± 0.342.56 ± 0.854.21 ± 0.34
Audio quality3.85 ± 0.261.69 ± 0.634.04 ± 0.40
Overall conversational naturalness3.96 ± 0.331.93 ± 0.824.06 ± 0.32

Dialogue continuation is an open-ended task with multiple plausible continuations. The recorded ground-truth continuation is one such example rather than a unique correct response, and it does not necessarily provide an upper bound on subjective ratings of coherence or turn-taking. We interpret the small differences in mean ratings as CSDLM receiving perceived-quality ratings close to those of recorded human conversations on the evaluated samples, rather than outperforming human conversations.

For audio quality, some ground-truth recordings from Fisher exhibit intermittent recording artifacts and indications of cross-channel leakage. In our evaluation, lower audio-quality ratings were concentrated in such recordings, bringing down the overall ground-truth mean.

AuthorsOpenReview ↗

Updated ablation results and listening study

We have completed the matched-condition ablations and incorporated the new results and updated listening study into the revised manuscript.

Matched-condition experiments (W1/Q1)

To evaluate the role of continuous audio representations, we compare CSDLM with a discrete-token variant, using Mimi RVQ [1] as the discrete representation. We replace the flow-matching head with a depth transformer [1] to predict audio tokens across the RVQ codebooks. To assess the text stream’s contribution to learning dialogue semantics and turn-taking, we also evaluate a variant that removes it while retaining continuous audio latents. All variants use the same backbone, Qwen3.5-2B-Base initialization, and training data, with 20K steps in each training stage.

Dialogue continuation

ModelHead params.GPT-Score ↑PPLIPUPauseGapOverlap
Ground Truth—3.828.349.3 (49.2%)4.3 (5.9%)5.6 (5.0%)7.8 (9.6%)
CSDLM47.6M2.636.579.9 (38.8%)6.0 (9.7%)5.4 (6.5%)7.7 (7.3%)
w/ Mimi RVQ645.9M2.307.098.8 (37.0%)2.3 (3.5%)3.9 (3.6%)10.6 (15.4%)
w/o text47.6M1.09266.231.5 (2.7%)1.6 (20.1%)0.6 (10.0%)0.1 (0.1%)

Text-conditioned generation

ModelHead params.Dialogue generation WER (%) ↓Dialogue generation SIM ↑Contextual TTS WER (%) ↓Contextual TTS SIM ↑Contextual TTS EmoSIM ↑TTS WER (%) ↓TTS SIM ↑
CSDLM47.6M10.610.5013.420.400.696.330.53
w/ Mimi RVQ645.9M14.100.4517.940.310.678.000.44

Compared with the Mimi RVQ variant, CSDLM achieves a higher GPT-Score on dialogue continuation, lower WER and higher SIM across all three text-conditioned generation tasks, and higher EmoSIM on contextual TTS, while using a substantially smaller audio prediction head. The gains in SIM and EmoSIM suggest that continuous-latent modeling helps capture fine-grained acoustic attributes, including speaker timbre and context-dependent emotional expression.

Removing the text stream substantially reduces GPT-Score and increases PPL, accompanied by a sharp reduction in speaking activity. These results support the text stream’s contribution to learning dialogue semantics and turn-taking under the evaluated training conditions. Our semantic findings are consistent with prior work showing that an internal text stream improves semantic coherence and accelerates convergence in speech language models [2].

The dialogue continuation ablation results are now included in Table 2 and discussed in Section 3.6. Results on the text-conditioned generation tasks, full experimental details, and further analysis are provided in Appendix C.

Updated listening study (W2/Q2)

We have expanded the listening study from 9 to 12 participants using the same evaluation protocol. The table below reports the updated means and 95% confidence intervals.

Evaluation criterionGround TruthdGSLMCSDLM
Semantic coherence4.05 ± 0.301.91 ± 0.684.17 ± 0.34
Turn-taking naturalness4.15 ± 0.383.04 ± 0.524.26 ± 0.30
Speaker similarity4.56 ± 0.182.02 ± 0.694.44 ± 0.24
Emotional coherence4.31 ± 0.272.48 ± 0.624.22 ± 0.30
Audio quality3.93 ± 0.261.79 ± 0.494.16 ± 0.31
Overall conversational naturalness4.01 ± 0.241.91 ± 0.584.09 ± 0.25

The overall trend remains unchanged: CSDLM receives substantially higher mean ratings than dGSLM across all six criteria, with ratings close to those of ground truth. The updated results are included in Table 3, and the confidence-interval calculation is described in Appendix B.1. We have also added all eight listening-study examples and the aggregate ratings to the Listening study samples subsection of our demo page.

References

[1] Défossez et al., Moshi: a speech-text foundation model for real-time dialogue, 2024.

[2] Chou et al., Quantifying the Generation Modality Gap in Speech-Text Language Models, 2026.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Updated Audio Demos

We thank all reviewers for their constructive feedback.

We have updated our audio demo page. In particular, the Dialogue Generation section includes overlapping speech and backchannels.

We invite the reviewers to listen to these updated examples alongside our responses.

AuthorsOpenReview ↗

Final statement and summary of manuscript updates

We thank the reviewers for their feedback, which helped us strengthen the experimental evidence and clarify the manuscript. As the discussion period concludes, we summarize the main additions.

CSDLM is a foundation model that supports dialogue continuation, dialogue generation, contextual TTS, and streaming TTS using the same checkpoint without task-specific fine-tuning. To achieve this, we propose an interleaved sequence of text tokens and continuous audio latents for two-channel dialogue modeling and investigate effective voice-prompt conditioning designs for autoregressive flow-matching speech generation.

Improvements during the rebuttal period:

  1. Representation and text-stream ablations. We compare continuous-latent modeling with a Mimi RVQ variant and evaluate a textless variant using the same backbone, pretrained initialization, training data, and training steps. Compared with Mimi RVQ, CSDLM achieves a higher GPT-Score, lower WER and higher SIM across all three text-conditioned tasks, and higher EmoSIM, while using a smaller audio prediction head (47.6M versus 645.9M parameters). Removing the text stream substantially reduces semantic coherence and speaking activity under these training conditions. The main results are included in Table 2 and Section 3.6, with additional results and analysis in Appendix C.

  2. Listening study. We conducted a blinded listening study with 12 participants evaluating continuations from 8 conversations, comparing CSDLM with dGSLM and ground-truth human continuations. CSDLM receives higher mean ratings than dGSLM and ratings close to ground truth across all six criteria, including semantic coherence and turn-taking naturalness. These results suggest that CSDLM can generate dialogue that closely resembles real human conversation. Updated results with 95% confidence intervals appear in Table 3, with evaluation details in Appendix B.1.

  3. Manuscript clarifications and demos. We clarify the distinction between dialogue continuation and transcript-conditioned synthesis, provide additional explanation of BOS sampling, and explain the rationale for the interleaving ratio. These revisions are highlighted in blue. All eight listening-study examples and ratings are available in the Listening study samples subsection of our demo page.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.