- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper empirically study the design choices of training full-duplex spoken language models, considering the dual-channel sequence layout, duplex training objective (whether to predict user sequence as well), text guidance granularity, synthetic vs. real dialogue data. The core contribution is revealing some empirical insights of how these factors affect the model performance.
Critical strengths
- The core motivation of the paper is good. As the full-duplex SLMs becomes more and more popular, a practical guideline of various design choices will be helpful for the community. Thus, the motivation of the paper is well-established.
- Overall, the paper is well-written and easy-to-follow.
Critical weaknesses
- Inconsistency of experimental setups: There seems to be some inconsistencies setups. For example, in RQ1, the chunk sizes considered are 1, 2, 4, and 8, yet in RQ2, chunk size of 2 is omitted without any clarification. In RQ3, there is a new metric, Pre-TOR, which is not considered in other RQs, and again, the rationale behind this is not well-justified. This makes the experiments inconsistent and makes the results less reliable.
- The step-aligned formulation does not make sense for me since the sequential conditioning of token prediction is not considered. While this is acknowledged by the author, I believe it is a critical comparison target that should be included, especially for a paper that aims to provide actionable insights of model design and training. For me, this omission is just mentioned and acknowledged without reasonable justification.
- In RQ2, the effect of joint modeling of both speakers is studied. However, the factor is not invesitgated, which weakens the insights of the paper. Since how to weigh the importance of assistant modeling and user modeling is also practically crucial, without an ablation of this factor makes the study incomplete.
- The comparison between real and synthetic training data is not sufficiently well controlled. Although the paper provides corpus-level statistics, the actual content and knowledge coverage of the two datasets are not matched. This raises the possibility that the model is simply exposed to more knowledge in one dataset than in the other. Given the substantial performance gap between the real and synthetic settings in Table 3, differences in dataset content could plausibly account for part of the observed improvement. Therefore, it is unclear whether the results demonstrate an inherent advantage of using synthetic data, rather than an advantage arising from differences in the amount or type of knowledge contained in the training data. The authors should better control for these confounding factors to isolate the effect of the data distribution itself.
- Since the gap shown in the comparison is not quite obvious, the paper should present some statistical test to ensure the findings are significant.
Questions for authors
I've listed my major concerns in the weakness part. One additional question is that can the author also include some human study. Since the paper heavily relies on existing benchmarks, where automatic evaluation dominates, it is unclear whether different designs can affect the human preference.
5 comments
Response Part 2
| Backbone | sc | λ=0 | λ=0.2 | λ=0.5 | λ=1 | λ=2 |
|---|---|---|---|---|---|---|
| GLM-4-Voice | 1 | 3.504 / 8.708 | 3.535 / 3.568 | 3.546 / 3.530 | 3.548 / 3.522 | 3.556 / 3.520 |
| 2 | 3.492 / 9.860 | 3.514 / 3.550 | 3.510 / 3.512 | 3.513 / 3.502 | 3.524 / 3.495 | |
| 4 | 3.486 / 8.799 | 3.495 / 3.540 | 3.497 / 3.516 | 3.507 / 3.509 | 3.518 / 3.504 | |
| 8 | 3.470 / 6.425 | 3.472 / 3.534 | 3.474 / 3.511 | 3.481 / 3.502 | 3.503 / 3.503 | |
| Kimi-Audio | 1 | 3.545 / 16.924 | 3.679 / 2.496 | 3.774 / 2.169 | 3.889 / 2.038 | 4.090 / 1.950 |
| 2 | 3.532 / 15.859 | 3.596 / 2.327 | 3.646 / 2.087 | 3.739 / 1.960 | 3.834 / 1.875 | |
| 4 | 3.516 / 14.822 | 3.559 / 2.146 | 3.593 / 1.964 | 3.637 / 1.875 | 3.692 / 1.812 | |
| 8 | 3.471 / 14.491 | 3.505 / 2.038 | 3.529 / 1.890 | 3.565 / 1.826 | 3.610 / 1.773 |
Two findings emerge. First, a small user-loss weight (λ=0.2) already achieves most of the reduction in user CE across all configurations, with diminishing gains at higher weights. Second, introducing user prediction consistently increases assistant CE, with a substantially larger cost on Kimi-Audio. At λ=1, this relative cost decreases as speech chunk size grows on both backbones.
To examine whether these prediction-level trade-offs translate into conversational performance, we also report the MTR and Post-TOR results across λ at sc=2.
| Backbone | λ | MTR ↑ | Post-TOR ↑ |
|---|---|---|---|
| GLM-4-Voice | 0 | 1.49 | 86% |
| 0.2 | 1.38 | 67% | |
| 0.5 | 1.46 | 71% | |
| 1 | 1.43 | 72% | |
| 2 | 1.47 | 64% | |
| Kimi-Audio | 0 | 1.33 | 81% |
| 0.2 | 1.12 | 48% | |
| 0.5 | 1.15 | 61% | |
| 1 | 1.13 | 67% | |
| 2 | 1.03 | 50% |
At sc=2, assistant-only training (λ=0) achieves the highest MTR and Post-TOR on both backbones. Among positive λ values, λ=1 achieves the highest Post-TOR, but still underperforms assistant-only training. Thus, improving user-speech prediction does not necessarily improve dialogue quality or interruption recovery.
Overall, these results reveal a trade-off between user- and assistant-speech modeling. A small positive λ substantially improves user prediction, but no positive weight consistently improves assistant-side conversational performance. The appropriate λ therefore depends on the target capability and backbone, rather than having a universally optimal value.
W4 · Confounding factors in real-versus-synthetic data
We agree that differences in dialogue content and knowledge coverage can contribute to the observed QA gaps. We would like to clarify that RQ4 compares dialogue data construction strategies, rather than isolating the effect of speech synthesis alone.
Real corpora such as Fisher [1] and CANDOR [2] capture spontaneous human interactions, preserving natural pauses, overlaps, and turn-taking patterns. Synthetic corpora such as Behavior-SD [3] and MultiTalkPT [4] can instead be deliberately constructed around target conversational tasks and interaction behaviors, potentially providing more task-relevant content and knowledge-based supervision. Differences in dialogue content and interaction patterns are therefore part of the corpus-level comparison we aim to study.
We agree that resynthesizing Fisher transcripts would help isolate acoustic effects. However, preserving Fisher's content would not introduce the task-oriented supervision or deliberately constructed interaction scenarios available in synthetic corpora. Such a control would therefore address a complementary question.
To examine the robustness of our findings, we added a second real–synthetic corpus pair (CANDOR vs. MultiTalkPT) and reran Fisher vs. Behavior-SD under matched training budgets. We excluded 322 CANDOR conversations overlapping with the MTR and FDB evaluation sets before constructing the training subset. Each run uses 700 optimizer updates, with less than 0.01% difference in effective target speech tokens within each pair. Both backbones use the same duplex adaptation recipe. Results below are averaged over four decoding seeds and reported as real → synthetic.
| Corpus pair | Backbone | MTR ↑ | Post-TOR ↑ | OAB Speech Acc. ↑ |
|---|---|---|---|---|
| Fisher → Behavior-SD | GLM-4-Voice | 2.21 → 1.60 | 69.1% → 65.4% | 3.30% → 16.26% |
| Fisher → Behavior-SD | Kimi-Audio | 1.93 → 1.39 | 76.4% → 54.4% | 4.82% → 7.64% |
| CANDOR → MultiTalkPT | GLM-4-Voice | 2.10 → 2.26 | 77.8% → 92.6% | 3.19% → 10.61% |
| CANDOR → MultiTalkPT | Kimi-Audio | 1.86 → 2.15 | 58.1% → 99.0% | 5.67% → 40.29% |
The QA trend is consistent across both corpus pairs and backbones: synthetic-trained models achieve higher OAB speech accuracy in all four comparisons. However, the effects on MTR and Post-TOR reverse across corpus pairs. Fisher outperforms Behavior-SD, whereas MultiTalkPT outperforms CANDOR on both backbones.
Response Part 3
These results suggest that QA performance is sensitive to corpus composition and task coverage, rather than demonstrating an inherent advantage of synthetic speech itself. Meanwhile, dialogue quality and turn-taking behavior depend strongly on the particular corpus. We will revise RQ4 to make this distinction explicit and qualify the conclusions accordingly.
W5 · Statistical significance of the comparisons
We agree that statistical testing can further strengthen the reliability of our findings. We therefore conducted two-sided paired permutation tests with Holm correction (adjusted p < 0.05) for the highlighted findings across RQ1–RQ4. Each finding involves multiple comparisons across different backbones, speech chunk sizes, or evaluation metrics, which we tested separately using matched evaluation samples.
Overall, 91 out of 130 tests (70%) are statistically significant. Importantly, all 13 findings with testable metrics have statistically significant supporting results: six have all tests significant, while the remaining seven contain both significant and non-significant tests. None has all tests non-significant. The remaining CE-only finding could not be tested because only aggregate validation losses were available.
The non-significant cases mainly occur in specific configurations, such as the MTR difference between step-aligned sc=4 and sc=8 on both backbones (RQ1), the MTR changes under joint training on GLM-4-Voice (RQ2), and the turn- versus chunk-level MTR difference under GLM Default (RQ3). Some Post-TOR and latency differences also do not reach significance, suggesting configuration-dependent effects rather than uniform improvements. All tested comparisons supporting the original RQ4 findings are significant.
Overall, these results provide broad statistical support for our main empirical findings. Since many findings describe general trends or configuration-dependent effects rather than universal improvements, individual non-significant comparisons do not necessarily undermine the overall conclusions. We will incorporate the statistical results and corresponding refinements into the revised paper.
Q1 · Human evaluation
We use established full-duplex benchmarks, including MTR-DuplexBench [5] and Full-Duplex-Bench [6]. Core turn-taking metrics, such as take-over rates and response latency, are computed automatically without requiring subjective human judgments. For dialogue quality, MTR follows the LLM-based evaluation pipeline of TurnGuide [7], whose alignment with human judgments has been empirically validated through listening studies. We therefore did not conduct a separate human study, while acknowledging that direct human evaluation could provide additional evidence of perceptual quality and human preference.
In the End
We will incorporate the additional experiments, revised interpretations, and corresponding limitations into the updated paper.
References
[1] Cieri, Christopher, et al. "The Fisher Corpus: A Resource for the Next Generations of Speech-to-Text." Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC '04), European Language Resources Association, 2004.
[2] Reece, Andrew, et al. "The CANDOR Corpus: Insights from a Large Multimodal Dataset of Naturalistic Conversation." Science Advances, vol. 9, no. 13, 2023, eadf3197.
[3] Lee, Sehun, et al. "Behavior-SD: Behaviorally Aware Spoken Dialogue Generation with Large Language Models." Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Association for Computational Linguistics, 2025, pp. 9574–9593.
[4] Wang, Ke, et al. "MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation." arXiv preprint arXiv:2609.36903, 2026.
[5] Zhang, He, et al. "MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models." Findings of the Association for Computational Linguistics: ACL 2026, Association for Computational Linguistics, 2026, pp. 5334–5351.
[6] Lin, Guan-Ting, et al. "Full-Duplex-Bench: A Benchmark to Evaluate Full-Duplex Spoken Dialogue Models on Turn-Taking Capabilities." arXiv preprint arXiv:2503.04721, 2025.
[7] Cui, Wenqian, et al. "TurnGuide: Enhancing Meaningful Full Duplex Spoken Interactions via Dynamic Turn-Level Text-Speech Interleaving." Proceedings of Interspeech 2026 [Long Track], 2026, pp. 5962–5970.
Response Part 1
Thank you for the constructive feedback and for recognizing the motivation and practical value of our study. We appreciate the concerns about experimental consistency and have conducted additional experiments to address them.
W1 · Consistency of experimental settings
We agree that the rationale for selecting different settings across RQs was not sufficiently explained. In RQ2, we originally selected speech chunk sizes of 1, 4, and 8 to cover a range of granularities while limiting computational cost. We have now added the missing sc=2 experiments for both backbones, comparing assistant-only training (λ=0) with joint user–assistant training (λ=1).
Values are reported as assistant-only (λ=0) → joint training (λ=1). All non-percentage metrics are rounded to two decimal places; percentages are rounded to whole numbers.
The new results are consistent with the main RQ2 observations: joint training increases assistant CE and reduces post-interruption latency on both backbones, while its effects on dialogue quality and Post-TOR remain configuration-dependent. Including sc=2, assistant CE increases in all eight backbone–chunk-size comparisons, and latency decreases in seven of eight.
Regarding Pre-TOR, this metric is defined in Section 4.3 and reported for all four RQs in Appendix G (Tables 9, 12, 15, and 18). We emphasize it in RQ3 because text-guidance granularity introduces a particularly relevant trade-off between yielding during an interruption and responding afterward. For example, on Kimi-Audio, moving from chunk- to turn-level guidance increases Pre-TOR from 5% to 14%, while reducing post-interruption latency from 1.47 to 1.25 seconds. We will clarify these choices in the revised paper.
W2 · Sequential conditioning in step-aligned prediction
We appreciate this concern and agree that sequential conditioning is an important architectural choice. Our original step-aligned implementation uses parallel prediction heads, which do not condition on preceding tokens within the same speech chunk. We therefore added within-chunk autoregressive (AR) prediction, where a lightweight causal decoder predicts assistant tokens sequentially, conditioning each prediction on preceding tokens within the chunk. We evaluated this at sc ∈ {2, 4, 8} on both backbones.
The following table summarizes the results. Each entry reports MTR / Post-TOR.
Within-chunk AR prediction improves Post-TOR in five of six comparisons, but the gains are generally modest and do not consistently improve MTR. Importantly, interleaved training still achieves higher MTR and Post-TOR than step-aligned AR across all six configurations.
These results suggest that the observed advantage of interleaving is not solely explained by the absence of within-chunk sequential conditioning in our original implementation. Nevertheless, our conclusions remain specific to the evaluated step-aligned architectures, rather than all possible implementations.
W3 · Effect of joint-training loss weight λ
We agree that balancing user and assistant prediction is an important design choice. We therefore extended RQ2 with a sweep over λ ∈ {0, 0.2, 0.5, 1, 2}, covering all four speech chunk sizes and both backbones.
The table below reports assistant CE / user CE for each configuration. Lower values indicate better prediction, and CE values should be compared within each backbone.