acceptodds
Under review as a conference paper at ICLR 2027

What Makes a Good Full-Duplex SLM? A Systematic Study of Architecture, Objectives, and Data

Abstract

Full-Duplex Speech Language Models (FD-SLMs) model user and assistant speech as separate channels, allowing a spoken assistant to listen while speaking and handle overlapping speech. However, existing systems combine different architectures, backbones, objectives, and data recipes, which makes their design choices difficult to compare. We conduct controlled experiments on four axes—dual-channel sequence layout and speech granularity, duplex training objectives, text guidance granularity, and real versus synthetic duplex data—using a shared pipeline and two open SLM backbones. Interleaved training improves mean dialogue quality and interruption recovery over our step-aligned implementation across both backbones and all tested speech chunk sizes. Joint user–assistant prediction with equal loss weights increases assistant-speech cross-entropy and usually reduces post-interruption latency, but its effects on dialogue quality and post-interruption recovery rate depend on the backbone and chunk size. Turn-level text guidance favors mean dialogue quality and faster post-interruption responses, whereas chunk-level guidance makes models easier to interrupt. Synthetic dialogue improves spoken QA on GLM-4-Voice and post-interruption latency and response rating on both backbones, but reduces post-interruption recovery rate; we also observe text-channel collapse in the synthetic-trained Kimi-Audio checkpoint. These findings provide actionable insights into selecting architectures, training objectives, and dialogue data to balance response quality, generation stability, and turn-taking behavior.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Speech Language Model, Full Duplex Dialogue Modeling, Systematic Study, Controlled Analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xsPEOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper empirically study the design choices of training full-duplex spoken language models, considering the dual-channel sequence layout, duplex training objective (whether to predict user sequence as well), text guidance granularity, synthetic vs. real dialogue data. The core contribution is revealing some empirical insights of how these factors affect the model performance.

Critical strengths
  1. The core motivation of the paper is good. As the full-duplex SLMs becomes more and more popular, a practical guideline of various design choices will be helpful for the community. Thus, the motivation of the paper is well-established.
  2. Overall, the paper is well-written and easy-to-follow.
Critical weaknesses
  1. Inconsistency of experimental setups: There seems to be some inconsistencies setups. For example, in RQ1, the chunk sizes considered are 1, 2, 4, and 8, yet in RQ2, chunk size of 2 is omitted without any clarification. In RQ3, there is a new metric, Pre-TOR, which is not considered in other RQs, and again, the rationale behind this is not well-justified. This makes the experiments inconsistent and makes the results less reliable.
  2. The step-aligned formulation does not make sense for me since the sequential conditioning of token prediction is not considered. While this is acknowledged by the author, I believe it is a critical comparison target that should be included, especially for a paper that aims to provide actionable insights of model design and training. For me, this omission is just mentioned and acknowledged without reasonable justification.
  3. In RQ2, the effect of joint modeling of both speakers is studied. However, the factor λ\lambda is not invesitgated, which weakens the insights of the paper. Since how to weigh the importance of assistant modeling and user modeling is also practically crucial, without an ablation of this factor makes the study incomplete.
  4. The comparison between real and synthetic training data is not sufficiently well controlled. Although the paper provides corpus-level statistics, the actual content and knowledge coverage of the two datasets are not matched. This raises the possibility that the model is simply exposed to more knowledge in one dataset than in the other. Given the substantial performance gap between the real and synthetic settings in Table 3, differences in dataset content could plausibly account for part of the observed improvement. Therefore, it is unclear whether the results demonstrate an inherent advantage of using synthetic data, rather than an advantage arising from differences in the amount or type of knowledge contained in the training data. The authors should better control for these confounding factors to isolate the effect of the data distribution itself.
  5. Since the gap shown in the comparison is not quite obvious, the paper should present some statistical test to ensure the findings are significant.
Questions for authors

I've listed my major concerns in the weakness part. One additional question is that can the author also include some human study. Since the paper heavily relies on existing benchmarks, where automatic evaluation dominates, it is unclear whether different designs can affect the human preference.

5 comments
AuthorsOpenReview ↗

Response Part 1

Thank you for the constructive feedback and for recognizing the motivation and practical value of our study. We appreciate the concerns about experimental consistency and have conducted additional experiments to address them.

W1 · Consistency of experimental settings

We agree that the rationale for selecting different settings across RQs was not sufficiently explained. In RQ2, we originally selected speech chunk sizes of 1, 4, and 8 to cover a range of granularities while limiting computational cost. We have now added the missing sc=2 experiments for both backbones, comparing assistant-only training (λ=0) with joint user–assistant training (λ=1).

BackbonescAssistant CE ↓MTR ↑Post-TOR ↑Latency (s) ↓
GLM-4-Voice13.50 → 3.551.28 → 1.2877% → 70%1.33 → 1.31
2 (New)3.49 → 3.511.49 → 1.4386% → 72%1.45 → 1.33
43.49 → 3.511.46 → 1.5167% → 66%1.75 → 1.52
83.47 → 3.481.46 → 1.5875% → 76%1.82 → 1.44
Kimi-Audio13.54 → 3.891.34 → 1.0162% → 76%0.75 → 0.98
2 (New)3.53 → 3.741.33 → 1.1381% → 67%0.70 → 0.54
43.52 → 3.641.35 → 1.1068% → 65%0.91 → 0.46
83.47 → 3.571.28 → 1.1975% → 88%1.15 → 0.37

Values are reported as assistant-only (λ=0) → joint training (λ=1). All non-percentage metrics are rounded to two decimal places; percentages are rounded to whole numbers.

The new results are consistent with the main RQ2 observations: joint training increases assistant CE and reduces post-interruption latency on both backbones, while its effects on dialogue quality and Post-TOR remain configuration-dependent. Including sc=2, assistant CE increases in all eight backbone–chunk-size comparisons, and latency decreases in seven of eight.

Regarding Pre-TOR, this metric is defined in Section 4.3 and reported for all four RQs in Appendix G (Tables 9, 12, 15, and 18). We emphasize it in RQ3 because text-guidance granularity introduces a particularly relevant trade-off between yielding during an interruption and responding afterward. For example, on Kimi-Audio, moving from chunk- to turn-level guidance increases Pre-TOR from 5% to 14%, while reducing post-interruption latency from 1.47 to 1.25 seconds. We will clarify these choices in the revised paper.

W2 · Sequential conditioning in step-aligned prediction

We appreciate this concern and agree that sequential conditioning is an important architectural choice. Our original step-aligned implementation uses parallel prediction heads, which do not condition on preceding tokens within the same speech chunk. We therefore added within-chunk autoregressive (AR) prediction, where a lightweight causal decoder predicts assistant tokens sequentially, conditioning each prediction on preceding tokens within the chunk. We evaluated this at sc ∈ {2, 4, 8} on both backbones.

The following table summarizes the results. Each entry reports MTR / Post-TOR.

BackbonescStep-Aligned ParallelStep-Aligned ARInterleaved
GLM-4-Voice20.506 / 42.7%0.568 / 46.5%1.491 / 86.0%
40.254 / 34.0%0.268 / 34.3%1.462 / 67.1%
80.226 / 32.4%0.251 / 37.2%1.461 / 75.3%
Kimi-Audio20.297 / 33.8%0.293 / 26.6%1.334 / 81.0%
40.198 / 34.9%0.198 / 38.6%1.349 / 68.4%
80.215 / 50.0%0.212 / 53.1%1.278 / 74.6%

Within-chunk AR prediction improves Post-TOR in five of six comparisons, but the gains are generally modest and do not consistently improve MTR. Importantly, interleaved training still achieves higher MTR and Post-TOR than step-aligned AR across all six configurations.

These results suggest that the observed advantage of interleaving is not solely explained by the absence of within-chunk sequential conditioning in our original implementation. Nevertheless, our conclusions remain specific to the evaluated step-aligned architectures, rather than all possible implementations.

W3 · Effect of joint-training loss weight λ

We agree that balancing user and assistant prediction is an important design choice. We therefore extended RQ2 with a sweep over λ ∈ {0, 0.2, 0.5, 1, 2}, covering all four speech chunk sizes and both backbones.

The table below reports assistant CE / user CE for each configuration. Lower values indicate better prediction, and CE values should be compared within each backbone.

AuthorsOpenReview ↗

Response Part 2

Backbonescλ=0λ=0.2λ=0.5λ=1λ=2
GLM-4-Voice13.504 / 8.7083.535 / 3.5683.546 / 3.5303.548 / 3.5223.556 / 3.520
23.492 / 9.8603.514 / 3.5503.510 / 3.5123.513 / 3.5023.524 / 3.495
43.486 / 8.7993.495 / 3.5403.497 / 3.5163.507 / 3.5093.518 / 3.504
83.470 / 6.4253.472 / 3.5343.474 / 3.5113.481 / 3.5023.503 / 3.503
Kimi-Audio13.545 / 16.9243.679 / 2.4963.774 / 2.1693.889 / 2.0384.090 / 1.950
23.532 / 15.8593.596 / 2.3273.646 / 2.0873.739 / 1.9603.834 / 1.875
43.516 / 14.8223.559 / 2.1463.593 / 1.9643.637 / 1.8753.692 / 1.812
83.471 / 14.4913.505 / 2.0383.529 / 1.8903.565 / 1.8263.610 / 1.773

Two findings emerge. First, a small user-loss weight (λ=0.2) already achieves most of the reduction in user CE across all configurations, with diminishing gains at higher weights. Second, introducing user prediction consistently increases assistant CE, with a substantially larger cost on Kimi-Audio. At λ=1, this relative cost decreases as speech chunk size grows on both backbones.

To examine whether these prediction-level trade-offs translate into conversational performance, we also report the MTR and Post-TOR results across λ at sc=2.

BackboneλMTR ↑Post-TOR ↑
GLM-4-Voice01.4986%
0.21.3867%
0.51.4671%
11.4372%
21.4764%
Kimi-Audio01.3381%
0.21.1248%
0.51.1561%
11.1367%
21.0350%

At sc=2, assistant-only training (λ=0) achieves the highest MTR and Post-TOR on both backbones. Among positive λ values, λ=1 achieves the highest Post-TOR, but still underperforms assistant-only training. Thus, improving user-speech prediction does not necessarily improve dialogue quality or interruption recovery.

Overall, these results reveal a trade-off between user- and assistant-speech modeling. A small positive λ substantially improves user prediction, but no positive weight consistently improves assistant-side conversational performance. The appropriate λ therefore depends on the target capability and backbone, rather than having a universally optimal value.

W4 · Confounding factors in real-versus-synthetic data

We agree that differences in dialogue content and knowledge coverage can contribute to the observed QA gaps. We would like to clarify that RQ4 compares dialogue data construction strategies, rather than isolating the effect of speech synthesis alone.

Real corpora such as Fisher [1] and CANDOR [2] capture spontaneous human interactions, preserving natural pauses, overlaps, and turn-taking patterns. Synthetic corpora such as Behavior-SD [3] and MultiTalkPT [4] can instead be deliberately constructed around target conversational tasks and interaction behaviors, potentially providing more task-relevant content and knowledge-based supervision. Differences in dialogue content and interaction patterns are therefore part of the corpus-level comparison we aim to study.

We agree that resynthesizing Fisher transcripts would help isolate acoustic effects. However, preserving Fisher's content would not introduce the task-oriented supervision or deliberately constructed interaction scenarios available in synthetic corpora. Such a control would therefore address a complementary question.

To examine the robustness of our findings, we added a second real–synthetic corpus pair (CANDOR vs. MultiTalkPT) and reran Fisher vs. Behavior-SD under matched training budgets. We excluded 322 CANDOR conversations overlapping with the MTR and FDB evaluation sets before constructing the training subset. Each run uses 700 optimizer updates, with less than 0.01% difference in effective target speech tokens within each pair. Both backbones use the same duplex adaptation recipe. Results below are averaged over four decoding seeds and reported as real → synthetic.

Corpus pairBackboneMTR ↑Post-TOR ↑OAB Speech Acc. ↑
Fisher → Behavior-SDGLM-4-Voice2.21 → 1.6069.1% → 65.4%3.30% → 16.26%
Fisher → Behavior-SDKimi-Audio1.93 → 1.3976.4% → 54.4%4.82% → 7.64%
CANDOR → MultiTalkPTGLM-4-Voice2.10 → 2.2677.8% → 92.6%3.19% → 10.61%
CANDOR → MultiTalkPTKimi-Audio1.86 → 2.1558.1% → 99.0%5.67% → 40.29%

The QA trend is consistent across both corpus pairs and backbones: synthetic-trained models achieve higher OAB speech accuracy in all four comparisons. However, the effects on MTR and Post-TOR reverse across corpus pairs. Fisher outperforms Behavior-SD, whereas MultiTalkPT outperforms CANDOR on both backbones.

AuthorsOpenReview ↗

Response Part 3

These results suggest that QA performance is sensitive to corpus composition and task coverage, rather than demonstrating an inherent advantage of synthetic speech itself. Meanwhile, dialogue quality and turn-taking behavior depend strongly on the particular corpus. We will revise RQ4 to make this distinction explicit and qualify the conclusions accordingly.

W5 · Statistical significance of the comparisons

We agree that statistical testing can further strengthen the reliability of our findings. We therefore conducted two-sided paired permutation tests with Holm correction (adjusted p < 0.05) for the highlighted findings across RQ1–RQ4. Each finding involves multiple comparisons across different backbones, speech chunk sizes, or evaluation metrics, which we tested separately using matched evaluation samples.

Overall, 91 out of 130 tests (70%) are statistically significant. Importantly, all 13 findings with testable metrics have statistically significant supporting results: six have all tests significant, while the remaining seven contain both significant and non-significant tests. None has all tests non-significant. The remaining CE-only finding could not be tested because only aggregate validation losses were available.

The non-significant cases mainly occur in specific configurations, such as the MTR difference between step-aligned sc=4 and sc=8 on both backbones (RQ1), the MTR changes under joint training on GLM-4-Voice (RQ2), and the turn- versus chunk-level MTR difference under GLM Default (RQ3). Some Post-TOR and latency differences also do not reach significance, suggesting configuration-dependent effects rather than uniform improvements. All tested comparisons supporting the original RQ4 findings are significant.

Overall, these results provide broad statistical support for our main empirical findings. Since many findings describe general trends or configuration-dependent effects rather than universal improvements, individual non-significant comparisons do not necessarily undermine the overall conclusions. We will incorporate the statistical results and corresponding refinements into the revised paper.

Q1 · Human evaluation

We use established full-duplex benchmarks, including MTR-DuplexBench [5] and Full-Duplex-Bench [6]. Core turn-taking metrics, such as take-over rates and response latency, are computed automatically without requiring subjective human judgments. For dialogue quality, MTR follows the LLM-based evaluation pipeline of TurnGuide [7], whose alignment with human judgments has been empirically validated through listening studies. We therefore did not conduct a separate human study, while acknowledging that direct human evaluation could provide additional evidence of perceptual quality and human preference.

In the End

We will incorporate the additional experiments, revised interpretations, and corresponding limitations into the updated paper.

References

[1] Cieri, Christopher, et al. "The Fisher Corpus: A Resource for the Next Generations of Speech-to-Text." Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC '04), European Language Resources Association, 2004.

[2] Reece, Andrew, et al. "The CANDOR Corpus: Insights from a Large Multimodal Dataset of Naturalistic Conversation." Science Advances, vol. 9, no. 13, 2023, eadf3197.

[3] Lee, Sehun, et al. "Behavior-SD: Behaviorally Aware Spoken Dialogue Generation with Large Language Models." Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Association for Computational Linguistics, 2025, pp. 9574–9593.

[4] Wang, Ke, et al. "MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation." arXiv preprint arXiv:2609.36903, 2026.

[5] Zhang, He, et al. "MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models." Findings of the Association for Computational Linguistics: ACL 2026, Association for Computational Linguistics, 2026, pp. 5334–5351.

[6] Lin, Guan-Ting, et al. "Full-Duplex-Bench: A Benchmark to Evaluate Full-Duplex Spoken Dialogue Models on Turn-Taking Capabilities." arXiv preprint arXiv:2503.04721, 2025.

[7] Cui, Wenqian, et al. "TurnGuide: Enhancing Meaningful Full Duplex Spoken Interactions via Dynamic Turn-Level Text-Speech Interleaving." Proceedings of Interspeech 2026 [Long Track], 2026, pp. 5962–5970.

Reviewer_xoUCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines four important but challenging research questions in training full-duplex speech models: sequence layout and speech chunk size, assistant-only versus joint user-assistant prediction, text guidance granularity, and real versus synthetic training data. It compares these design choices using two pretrained backbones and analyzes their effects on conversational quality and interaction behavior.

Critical strengths
  • [Strength 1] Practical relevance of the design choices. Having followed nearly all developments in full-duplex speech modeling since dGSLM and Moshi and gained substantial hands-on experience training these models, I find that the paper handles several practical design choices that researchers in this area genuinely need to consider.

  • [Strength 2] Evaluation across two backbones. The study uses two backbone models rather than one, providing evidence of whether the observed trends are consistent across models.

Critical weaknesses
  • [Weakness 1] Low absolute conversational performance. The low MTR scores and FDB post-interruption relevance ratings raise concerns about the models’ ability to sustain meaningful, contextually appropriate conversations, while the low OAB accuracy also indicates limited spoken QA performance. I recognize that the goal is to compare design choices rather than develop a state-of-the-art full-duplex model. Nevertheless, the reported scores suggest substantial difficulties with basic conversational competence. This makes it unclear whether the observed differences in dialogue dynamics reflect improvements in meaningful interaction, and whether the findings would extend to models that reliably sustain coherent conversations.

  • [Weakness 2] Confounded real-versus-synthetic comparison. Fisher and Behavior-SD differ in dialogue content, speakers, turn construction, and training update counts, while recording and transcription quality may introduce further confounds. Although the paper acknowledges several of these limitations, the comparison does not clearly isolate the effect of real versus synthetic speech. A stronger control would preserve dialogue content across conditions, for example by resynthesizing Fisher transcripts or constructing paired real and synthetic versions from a higher-quality conversational corpus. Human listening evaluation would also help establish whether the observed differences translate into perceptible improvements in speech quality, conversational naturalness, and overall satisfaction.

Questions for authors

I would appreciate the authors’ responses to the concerns raised above, along with additional experiments to address them. While additional experiments addressing W2 would be valuable, I would particularly appreciate the authors’ perspective on how to interpret the comparisons of dialogue dynamics given the very low conversational quality scores. Specifically, to what extent can these comparisons support meaningful conclusions when it is unclear whether the models can reliably sustain coherent conversations?

3 comments
AuthorsOpenReview ↗

Response Part 1

Thank you for the thoughtful feedback and for recognizing the practical relevance of our design choices and the value of evaluating two backbones.

W1 · Low absolute conversational performance

We understand the concern that the relatively low MTR, post-interruption relevance, and OAB scores may limit the generalizability of our findings to more capable conversational models. However, low absolute performance does not necessarily imply that the relative effects of design choices would disappear in stronger models.

We chose Fisher [1] as our primary training corpus because it provides about 2,000 hours of two-channel English conversations with natural turn-taking and overlap. Its roughly 1,518-hour training split allowed us to compare more configurations across two backbones within a manageable computational budget. Fisher has also been widely used in full-duplex modeling, including dGSLM [2], SyncLLM [3], NTPP [4], and TurnGuide [5], making it a well-established corpus for studying duplex interactions.

Training at the scale of systems such as Moshi [6], which used 7 million hours of single-stream audio for pretraining and over 20,000 hours of synthetic interactions for instruction tuning, would substantially limit the breadth of our comparisons. Nevertheless, we expect larger and more diverse training corpora to improve absolute performance. For example, MultiTalk [7] held the preceding training stage fixed and found that increasing the proportion of MultiTalkFT data from 0% to 100% improved its MultiTalkBench score from 7.95 to 13.15.

The low OAB scores also partly reflect a mismatch between casual conversational training data and factual spoken QA evaluation, potentially compounded by forgetting during adaptation [8]. Our additional experiments further demonstrate the importance of corpus choice: under matched training budgets and the same duplex adaptation recipe, Kimi-Audio achieves 7.92% Llama Questions speech accuracy with CANDOR versus 50.83% with MultiTalkPT (see W2 for the full comparison).

While generalization to substantially stronger models requires further validation, our controlled comparisons across two distinct backbones provide evidence of how individual design choices affect duplex learning, without assuming that low absolute performance invalidates these relative findings.

W2 · Real-versus-synthetic comparison and confounding factors

We agree that Fisher and Behavior-SD differ in multiple aspects, and that our original comparison cannot isolate the effect of speech synthesis alone. However, we would like to clarify that our intention is for RQ4 to compare different dialogue data construction strategies, rather than real recordings against TTS-generated versions of identical content.

Real corpora such as Fisher and CANDOR capture spontaneous human conversations, preserving natural turn-taking, pauses, and overlapping speech. In contrast, synthetic corpora such as Behavior-SD and MultiTalkPT are deliberately constructed to support target conversational tasks and interaction behaviors, including instruction following, turn-taking, and interruption handling. These approaches differ not only in acoustic realization but also in dialogue content, knowledge coverage, and interaction patterns.

We appreciate the suggestion to resynthesize Fisher transcripts. Such an experiment would help isolate acoustic differences, but addresses a different question from ours. Preserving Fisher's conversational content would not introduce the knowledge-based supervision or deliberately designed interaction scenarios that synthetic corpora can provide. We therefore view paired resynthesis as a complementary control rather than a substitute for comparing corpus construction strategies.

To examine whether our observations hold beyond the original dataset pair, we added CANDOR vs. MultiTalkPT and reran Fisher vs. Behavior-SD under matched training budgets. The CANDOR and MultiTalkPT subsets contain 686.6 and 675.7 hours of dialogue, respectively. We excluded 322 CANDOR conversations overlapping with the MTR and FDB evaluation sets before constructing the training subset. Both backbones use the same interleaved layout, speech chunk size of 1, turn-level text guidance, and assistant-only objective. Each run uses 700 optimizer updates, with less than 0.01% difference in effective target speech tokens within each pair. MTR, Post-TOR, and QA results are averaged over four decoding seeds; text emission is measured in a separate evaluation pass. Values are reported as real → synthetic.

AuthorsOpenReview ↗

Response Part 2

Corpus pairBackboneMTR ↑Post-TOR ↑Llama Q. Speech Acc. ↑OAB Speech Acc. ↑Text Emission
Fisher → Behavior-SDGLM-4-Voice2.21 → 1.6069.1% → 65.4%5.42% → 29.42%3.30% → 16.26%89.08% → 90.75%
Fisher → Behavior-SDKimi-Audio1.93 → 1.3976.4% → 54.4%7.25% → 12.17%4.82% → 7.64%66.92% → 29.08%
CANDOR → MultiTalkPTGLM-4-Voice2.10 → 2.2677.8% → 92.6%4.08% → 16.25%3.19% → 10.61%74.33% → 94.00%
CANDOR → MultiTalkPTKimi-Audio1.86 → 2.1558.1% → 99.0%7.92% → 50.83%5.67% → 40.29%52.42% → 98.75%

Two findings emerge. First, synthetic-trained models consistently achieve higher OAB speech accuracy across both corpus pairs and backbones. Second, dialogue quality and interruption recovery are corpus-dependent. Fisher outperforms Behavior-SD on MTR and Post-TOR even after budget matching, whereas MultiTalkPT outperforms CANDOR on both backbones. Text emission also varies substantially: on Kimi-Audio, MultiTalkPT achieves 98.75% versus 52.42% for CANDOR, coinciding with its large QA gain. Budget matching also partially alleviates the previously observed text-channel collapse with Behavior-SD. These results support the observed QA trend while showing that interaction outcomes depend on the specific corpus rather than simply whether the speech is real or synthetic.

Regarding human evaluation, we use established benchmarks, including MTR-DuplexBench and Full-Duplex-Bench. Core turn-taking metrics, such as take-over rates and response latency, are computed automatically without requiring subjective human judgments. For dialogue quality, MTR follows the LLM-based evaluation pipeline of TurnGuide [5], whose alignment with human judgments has been empirically validated. We therefore consider these benchmarks appropriate for our controlled comparisons, while acknowledging that additional human evaluation could further strengthen the conclusions.

Q1 · Interpretation and generalizability of dialogue dynamics

As discussed in W1, we believe that low absolute conversational performance does not necessarily invalidate controlled comparisons. Our key argument is that full-duplex conversational ability involves two related but partially separable dimensions: semantic quality (what the model says) and turn-taking behavior (when it speaks, pauses, or yields).

Although low MTR scores indicate limited semantic competence, relative differences can still reveal improvements in dialogue quality under the same evaluation protocol. Meanwhile, limited semantic coherence does not necessarily prevent a model from exhibiting natural turn-taking behavior. For example, dGSLM [2] demonstrated naturalistic turn-taking patterns despite its limited semantic coherence. Thus, both dimensions remain meaningful to compare even when absolute conversational performance is relatively low.

As discussed in W1, scaling to larger and more diverse training corpora would likely improve absolute performance. However, our computational budget required a trade-off between training scale and the breadth of controlled comparisons. We therefore focus on identifying relative effects of design choices across two backbones, while recognizing that their transfer to substantially stronger models warrants further validation.

In the End

We will incorporate the additional experiments, revised interpretations, and corresponding limitations into the updated paper.

AuthorsOpenReview ↗

Response Part 3

Reference

[1] Cieri, Christopher, et al. "The Fisher Corpus: A Resource for the Next Generations of Speech-to-Text." Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC '04), European Language Resources Association, 2004.

[2] Nguyen, Tu Anh, et al. "Generative Spoken Dialogue Language Modeling." Transactions of the Association for Computational Linguistics, vol. 11, 2023, pp. 250–266.

[3] Veluri, Bandhav, et al. "Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents." Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2024, pp. 21390–21402.

[4] Wang, Qichao, et al. "NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction." Proceedings of the 42nd International Conference on Machine Learning, vol. 267, PMLR, 2025, pp. 63912–63928.

[5] Cui, Wenqian, et al. "TurnGuide: Enhancing Meaningful Full Duplex Spoken Interactions via Dynamic Turn-Level Text-Speech Interleaving." Proceedings of Interspeech 2026 [Long Track], 2026, pp. 5962–5970.

[6] Défossez, Alexandre, et al. "Moshi: A Speech-Text Foundation Model for Real-Time Dialogue." arXiv preprint arXiv:2410.00037, 2024.

[7] Wang, Ke, et al. "MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation." arXiv preprint arXiv:2609.36903, 2026.

[8] Hsiao, Chi-Yuan, et al. "Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models." Proceedings of Interspeech 2025, 2025, pp. 3234–3238.

Reviewer_oEbcOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper conduct controlled experiments on four axes for full-duplex SLM. Four axes includes dual-channel sequence layout and speech granularity, duplex training objectives, text guidance granularity, and real versus synthetic duplex data. The paper uses a share pipeline and two open SLM backbones (GLM-4-Voice and Kimi-Audio). Main findings includes interleaved training is better than step-aligned, assistant-speech CE is better than joint user-assistant prediction, turn-level text guidance favors dialogue quality, synthetic dialogue improve spoken QA. These findings provide some insights for selecting method for full-duplex SLM.

Critical strengths
  1. The paper presents a comprehensive Full-duplex SLM framework, including its model architecture, objective function, and a comparison of data, all within a relatively fair experimental setting.
  2. Full-duplex SLM is currently a popular research area, and this paper offers valuable insights for model selection.
Critical weaknesses
  1. The novelty is somewhat limited, as the chosen dimensions and the various methods employed are all quite standard.
  2. The overall performance—particularly the results for IQ dataset OAB—is rather low, falling short of a level that would be considered practically usable. Consequently, the conclusions drawn may not necessarily apply to more advanced models.
  3. The comparison between real and synthetic data is quite narrow, covering only two datasets.
Questions for authors
  1. Have you tried training using a combination of real and synthetic data? Does this lead to better results?
  2. Besides these dimensions, are there any other aspects that could be used for comparative analysis?
3 comments
AuthorsOpenReview ↗

Response Part 1

Thank you for the thoughtful feedback and for recognizing the value of the controlled comparisons.

W1 · Novelty and choice of settings

We agree that the individual techniques are standard practices for developing full duplex models. We deliberately chose them because they are decisions practitioners typically face when adapting a speech model to full duplex. Published systems often vary architecture, objective, and data together, so their results leave open which choice accounts for a given difference. Our main contribution is a systematic, controlled comparison of these design choices across two distinct backbones, identifying which findings generalize across models and which are configuration-dependent.

W2 · Absolute performance and transfer

We understand the reported OAB scores are not very competitive. We chose Fisher [1] as the main training corpus for the controlled comparisons. It provides about 2,000 hours of two-channel English telephone conversations with natural turn-taking and overlap. Using its roughly 1,518-hour training split kept the cost per configuration manageable, so we could compare more settings across both backbones. Fisher also has a history in full-duplex modeling. dGSLM [2], SyncLLM [3], NTPP [4], and TurnGuide [5] all used it in their training recipes.

Repeating training at the scale of well-trained backbones, such as Moshi [6], would sharply limit the number of comparisons we could run. Its recipe used 7 million hours of single-stream audio for pretraining and over 20,000 hours of synthetic interactions for instruction tuning. However, we expect larger, more diverse corpora to improve absolute performance. For example, MultiTalk [7] held the preceding stage fixed and found that increasing the share of MultiTalkFT from 0% to 100% raised its MultiTalkBench score from 7.95 to 13.15.

Another reason why OpenAudioBench (OAB) scores are relatively low is that the benchmark tests factual spoken QA, a task with little direct supervision in our dialogue adaptation. Limited exposure and possible forgetting during adaptation [8] could both contribute to low scores. To see how much the choice of dialogue corpus affects OAB, we added a comparison between real CANDOR and synthetic MultiTalkPT, using roughly 680 hours of dialogue per corpus and matching optimizer updates and target speech tokens. Under the same duplex adaptation recipe, Kimi-Audio trained on CANDOR achieved 7.92% speech accuracy on Llama Questions, and Kimi-Audio trained on MultiTalkPT achieved 50.83% (see W3 for both backbones and the full comparison). The size of this gap shows how sensitive absolute QA scores can be to the training corpus. Our design conclusions come from comparing configurations within the same backbone under controlled settings.

W3 · Additional real and synthetic corpora

To address the limited corpus coverage and training-budget differences in RQ4, we added a second real–synthetic pair (CANDOR vs. MultiTalkPT) and reran Fisher vs. Behavior-SD under matched training budgets (using same optmization steps). The CANDOR and MultiTalkPT subsets contain 686.6 and 675.7 hours of dialogue, respectively. We excluded 322 CANDOR conversations overlapping with the MTR and FDB evaluation sets before constructing the training subset. Both backbones use the same interleaved layout, speech chunk size of 1, turn-level text guidance, and assistant-only objective. Each run uses 700 optimizer updates, with less than 0.01% difference in effective target speech tokens within each pair. Results are averaged over four decoding seeds. Values are reported as real → synthetic. OAB speech accuracy averages three subsets equally; text emission measures the fraction of Llama Questions outputs containing non-padding text tokens.

Corpus pairBackboneMTR ↑Post-TOR ↑Llama Questions speech accuracy ↑OAB speech accuracy ↑Text emission rate
Fisher → Behavior-SDGLM-4-Voice2.21 → 1.6069.1% → 65.4%5.42% → 29.42%3.30% → 16.26%89.08% → 90.75%
Fisher → Behavior-SDKimi-Audio1.93 → 1.3976.4% → 54.4%7.25% → 12.17%4.82% → 7.64%66.92% → 29.08%
CANDOR → MultiTalkPTGLM-4-Voice2.10 → 2.2677.8% → 92.6%4.08% → 16.25%3.19% → 10.61%74.33% → 94.00%
CANDOR → MultiTalkPTKimi-Audio1.86 → 2.1558.1% → 99.0%7.92% → 50.83%5.67% → 40.29%52.42% → 98.75%
AuthorsOpenReview ↗

Response Part 2

Two findings emerge. First, synthetic-trained models consistently achieve higher OAB speech accuracy across both corpus pairs and backbones. Second, the effects on dialogue quality and interruption recovery are corpus-dependent. Fisher outperforms Behavior-SD on MTR and Post-TOR, even after budget matching, whereas MultiTalkPT outperforms CANDOR on both backbones. In particular, MultiTalkPT improves Post-TOR by 14.9 percentage points on GLM-4-Voice and 40.9 points on Kimi-Audio. These results suggest that turn-taking performance depends on the particular dialogue corpus rather than simply whether the speech is real or synthetic.

Text-channel behavior also varies substantially across corpora and training settings. On Kimi-Audio, the higher text emission rate with MultiTalkPT (98.75% vs. 52.42% on CANDOR) coincides with a large QA accuracy gain. Budget matching also partially alleviates the previously observed text-channel collapse with Behavior-SD, although its emission rate remains below Fisher's. These observations suggest that text-channel behavior may contribute to the QA differences, but do not isolate their cause. Overall, the additional experiments strengthen the evidence for the observed QA trend while showing that conclusions about dialogue quality and turn-taking require corpus-specific qualification.

Q1 · Mixing real and synthetic data

Yes. Appendix G (Tables 16–18) reports a 50% synthetic mixture in our original audio-duration-matched experiments. For brevity, we summarize the key results below, with the full results provided in the appendix.

BackboneTraining dataMTR ↑OAB Speech Acc. ↑Post-TOR ↑
GLM-4-VoiceFisher (Real)2.133.19%71%
50% Mixed2.2314.58%67%
Behavior-SD (Synthetic)1.5816.60%52%
Kimi-AudioFisher (Real)1.924.79%71%
50% Mixed1.950.72%16%
Behavior-SD (Synthetic)1.680.98%26%

On GLM-4-Voice, mixing achieved the highest mean MTR score, retained most of the synthetic-data QA gain, and partially recovered the Post-TOR lost with synthetic-only training. This suggests that mixing can balance dialogue quality, QA accuracy, and interruption recovery. However, the same mixture did not benefit Kimi-Audio. Although MTR remained comparable to real-only training, both QA accuracy and Post-TOR dropped below even the synthetic-only results. The QA degradation may be related to the text-channel instability observed with synthetic-trained Kimi-Audio.

Q2 · Other dimensions for comparative analysis

Several additional dimensions merit investigation. Data scale and composition, particularly the proportion of knowledge-based instruction data, could help determine how to improve QA without sacrificing turn-taking. Speech tokenization and user-stream fusion strategies may affect speech modeling efficiency and interaction quality, while decoding and turn-taking policies could influence response latency and interruption handling.

We prioritized four fundamental choices in duplex adaptation: sequence layout determines how the two speech streams are organized, training objectives determine which streams are supervised, text guidance controls when linguistic information becomes available, and corpus selection determines the interaction patterns encountered during training. Studying these choices within a shared pipeline provides a foundation for investigating the additional dimensions above.

In the End

We have addressed each point here and will incorporate the new results and corresponding changes into the revised paper.

AuthorsOpenReview ↗

Response Part 3

References

[1] Cieri, Christopher, et al. "The Fisher Corpus: A Resource for the Next Generations of Speech-to-Text." Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC '04), European Language Resources Association, 2004.

[2] Nguyen, Tu Anh, et al. "Generative Spoken Dialogue Language Modeling." Transactions of the Association for Computational Linguistics, vol. 11, 2023, pp. 250–66.

[3] Veluri, Bandhav, et al. "Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents." Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2024, pp. 21390–402.

[4] Wang, Qichao, et al. "NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction." Proceedings of the 42nd International Conference on Machine Learning, vol. 267, PMLR, 2025, pp. 63912–28.

[5] Cui, Wenqian, et al. "TurnGuide: Enhancing Meaningful Full Duplex Spoken Interactions via Dynamic Turn-Level Text-Speech Interleaving." Proceedings of Interspeech 2026 [Long Track], 2026, pp. 5962–70.

[6] Défossez, Alexandre, et al. "Moshi: A Speech-Text Foundation Model for Real-Time Dialogue." arXiv preprint arXiv:2410.00037 (2024).

[7] Wang, Ke, et al. "MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation." arXiv preprint arXiv:2609.36903 (2026).

[8] Hsiao, Chi-Yuan, et al. "Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models." Proceedings of Interspeech 2025, 2025, pp. 3234–38.

Comments on the paper

1 comment
Area_Chair_wdqvOpenReview ↗

Please reply to authors' rebuttal if you haven't done so

Dear Reviewers, As the author–reviewer discussion deadline approaches, I would like to remind you to review the authors’ rebuttal and indicate whether your concerns have been adequately addressed. If you have any remaining concerns or questions, please share them as soon as possible to leave enough time for discussion.

Thank you again for your time and effort. AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.