- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper handles whether full-duplex speech models can accurately track how far they have spoken when interrupted by a user. The authors introduce AnchorSpeech, a benchmark and training dataset for this capability, and propose Self-Listening, which feeds the model’s played speech back into its input. Experiments show that existing models struggle with this task, while the proposed model achieves substantially higher anchoring accuracy.
Critical strengths
-
The paper identifies an underexplored problem in full-duplex speech models and introduces a benchmark and a modeling approach to address it. The proposed model shows substantial improvements on the benchmark.
-
The authors also evaluate their model on Full-Duplex-Bench v1.5 to assess its broader full-duplex conversational capabilities.
Critical weaknesses
-
The claims around Self-Listening appear overstated. Moshi, PersonaPlex, SALMONN-omni, MiniCPM-o 4.5, and Raon-SpeechChat already incorporate representations of their own outputs into subsequent inputs while processing user speech. The claimed distinction is therefore playback alignment (mentioned in related work), not self-listening itself. The paper does not sufficiently establish what additional information this feedback provides over existing output histories or why it requires the proposed architecture.
-
The implementation of playback alignment is unclear. Feeding back speech when it is actually played requires synchronization with user-side playback, accounting for generation latency, network delays, and buffering. These details are missing. If the model simply receives generated speech intended for playback, its distinction from existing approaches is unclear.
-
The proposed model is trained and evaluated on similarly constructed data. While valid, this setting does not establish a structural advantage or generalization to realistic interactions and different data distributions.
-
Existing full-duplex models should be fine-tuned on the same data, including AnchorSpeech and InstructS2S, and compared under matched conditions to establish the benefit of Self-Listening. This comparison is essential to determine whether the gains come from the proposed feedback mechanism or task-specific training.
-
The evaluation lacks broader measures of conversational dynamics, semantic capabilities, and dialogue quality, such as the full Full-Duplex-Bench v1, VoiceBench, and human evaluation. Results on a subset from Full-Duplex-Bench v1.5 alone are insufficient to establish the model’s overall full-duplex conversational capabilities.
Overall, the paper raises a relevant question, but the current evidence does not sufficiently establish the novelty, necessity, or generalizability of the proposed approach.
Questions for authors
-
Could the authors clarify the precise distinction between Self-Listening and existing approaches? Despite repeated reading, I find the current explanation insufficient to identify the substantive architectural difference.
-
Could the authors demonstrate the benefit of this distinction by additionally training existing full-duplex models on InstructS2S-200K and AnchorSpeech and comparing their performance under matched conditions?
-
Could the authors evaluate the model on the full Full-Duplex-Bench v1 and semantic benchmarks such as OpenAudioBench or VoiceBench to establish its broader capabilities as a full-duplex speech model?
3 comments
Response to Reviewer 12sa (2)
W4 & Q2: Fair Comparison and Controlled Ablation
To isolate the contribution of the Self-Listening mechanism from that of task-specific training data, we have already conducted a controlled ablation study comparing 2-channel and 3-channel models in our original paper. Both models share the same Qwen2.5-Omni backbone and training data, including AnchorSpeech, with the primary difference being the presence of the Self-Listening channel. Despite being trained on the same data, the 2-channel model achieves only 7.8% anchoring accuracy on AnchorSpeech-test, whereas the 3-channel Self-Listening model achieves 73.0%. These results demonstrate that the improvement cannot be attributed solely to task-specific training data, highlighting the critical role of explicit playback-grounded feedback in achieving reliable anchoring.
We agree that fine-tuning existing full-duplex models on the same data would provide additional insights. However, retraining all these models is challenging given the computational resources and training costs involved. Moreover, these models differ substantially in their pretraining data, architectures, speech representations, and training strategies, making it difficult to ensure a strictly fair comparison even with identical fine-tuning data. Therefore, we adopted a controlled ablation with the same backbone and training configuration to more directly isolate the contribution of Self-Listening. We leave broader cross-architecture comparisons under unified training conditions for future work.
W5 & Q3: Broader Conversational Capabilities
(1) Additional evaluation on Full-Duplex-Bench v1.
Following the reviewer's suggestion, we further evaluated our model on the complete Full-Duplex-Bench v1, covering Pause Handling, Smooth Turn-taking, Backchannel, and User Interruption. As shown in Table 1, our model achieves the best performance on several Pause Handling and Backchannel metrics, while remaining competitive on Smooth Turn-taking. Since our training data primarily focuses on user interruptions and anchoring, without specifically incorporating False Handling and Model Interruption scenarios, some performance gaps in general interaction metrics are understandable. Overall, these results demonstrate that our model maintains competitive full-duplex conversational capabilities while acquiring the additional anchoring capability.
Table 2. Evaluation on Full-Duplex-Bench v1.
| Task | Dataset | Metric | PersonaPlex | dGSLM | Moshi | Freeze-Omni | 3-ch Self-Listening |
|---|---|---|---|---|---|---|---|
| Pause Handling | Synthetic (137) | TOR ↓ | 0.584 | 0.934 | 0.985 | 0.642 | 0.5474 |
| Pause Handling | Candor (216) | TOR ↓ | 0.662 | 0.935 | 0.980 | 0.481 | 0.5926 |
| Smooth Turn-taking | Candor (119) | TOR ↑ | 0.992 | 0.975 | 0.941 | 0.336 | 0.9664 |
| Smooth Turn-taking | Candor (119) | Latency (s) ↓ | 0.070 | 0.352 | 0.265 | 0.953 | 0.2642 |
| Backchannel | ICC (55) | TOR ↓ | 0.327 | 0.691 | 1.000 | 0.636 | 0.1455 |
| Backchannel | ICC (55) | Frequency (/s) ↑ | 0.025 | 0.015 | 0.001 | 0.001 | 0.0250 |
| Backchannel | ICC (55) | JSD ↓ | 0.649 | 0.934 | 0.957 | 0.997 | 0.8901 |
| User Interruption | Synthetic (200) | TOR ↑ | 1.000 | 0.917 | 1.000 | 0.867 | 0.9750 |
| User Interruption | Synthetic (200) | GPT Score (0–5) ↑ | 4.210 | 0.201 | 0.765 | 3.615 | 2.5436 |
| User Interruption | Synthetic (200) | Latency (s) ↓ | 0.400 | 2.531 | 0.257 | 1.409 | 0.3138 |
(2) Additional evaluation of semantic capabilities.
We further evaluated our model on three QA benchmarks: Llama Questions, WebQuestions, and TriviaQA. As shown in Table 2, our 3-channel Self-Listening model achieves accuracies of 69.0% and 37.9% on Llama Questions and TriviaQA, respectively, outperforming all evaluated full-duplex baselines, while remaining competitive on WebQuestions. These results indicate that our model retains strong general knowledge and question-answering capabilities while improving anchoring performance.
Table 3. QA performance compared with representative full-duplex models (accuracy, %, ↑).
| Dataset | Moshi | Freeze-Omni | PersonaPlex | 3-ch Self-Listening |
|---|---|---|---|---|
| Llama Questions | 54.5 | 56.2 | 61.7 | 69.0 |
| WebQuestions | 22.1 | 27.9 | 22.5 | 23.4 |
| TriviaQA | 16.7 | 28.5 | 22.8 | 37.9 |
Kindly Reminder
Thank you again for your time and thoughtful review. With the discussion period closing in one day, we would appreciate hearing whether our responses have addressed your concerns and whether any outstanding issues remain. If your concerns have been resolved, we would be grateful if you would consider updating your rating accordingly.
Response to Reviewer 12sa (1)
We thank the reviewer for recognizing the importance of the anchoring problem and for the valuable feedback and constructive suggestions. Below, we address the reviewer’s main concerns.
We thank the reviewer for the insightful discussion regarding the distinction between Self-Listening and existing full-duplex models. We agree that conditioning subsequent predictions on previously generated speech has already been adopted by models such as Moshi and MiniCPM-o. We would like to clarify that the key distinction lies in what information the feedback represents and what capabilities the model can acquire from it.
Existing models such as Moshi incorporate previously generated speech tokens into their autoregressive context to support continuous speech generation. However, their audio feedback reflects generation progress rather than actual playback progress. Due to speech synthesis and playback delays, the model's autoregressive context may already contain speech that has not yet been delivered to the user. Consequently, when an interruption occurs, the model cannot reliably determine the actual spoken boundary from its generation history alone, making it difficult to accurately track how far it has spoken. This limitation is also reflected in our experiments, where Moshi and MiniCPM-o 4.5 achieve anchoring accuracies of only 6.7% and 24.7%, respectively, on AnchorSpeech-test.
In contrast, Self-Listening continuously feeds the model's actual speaking progress back into the model through a dedicated feedback channel, independently of its internal generation history. By explicitly modeling the generated content and the actually played speech as separate streams, the model can distinguish between what it has generated and what the user has actually heard, even when speech generation and playback are asynchronous. Through targeted anchor-interruption training, the model learns to associate its generation history with the realized speech progress, allowing it to accurately identify how far its speech has reached the user and recover from the correct position when interrupted.
We thank the reviewer for raising this concern. Our implementation accounts for speech generation latency and audio buffering. Specifically, since our model is deployed locally, we can directly track the audio player's playback position. Generated speech is first placed in the playback buffer, and only audio segments that have actually been played are fed back into the Self-Listening channel. Unplayed audio is excluded, ensuring that the feedback remains aligned with actual playback progress, even when generation and playback are asynchronous.
Since our original experiments use local deployment, they do not involve network transmission delays. To address this concern, we conducted an additional robustness ablation by introducing random feedback delays of 0–400 ms into the Self-Listening channel. The anchoring accuracy remains between 70% and 75%, comparable to approximately 74% in the original setting. These results suggest that Self-Listening is robust to moderate feedback transmission delays and maintains stable anchoring performance under the tested conditions.
To the best of our knowledge, there is currently no established benchmark explicitly evaluating whether a model can accurately track its own speaking progress during interruptions. As the first work to systematically investigate this problem, we had to construct both the training and evaluation datasets ourselves, making some similarity in their data distributions difficult to avoid. We acknowledge this limitation and hope that future work will further investigate anchoring in diverse real-world conversational settings.
Nevertheless, to better evaluate generalization to realistic scenarios, we additionally constructed a new held-out Semantic Anchor-Interruption Test Set, following the procedure in Section 4.2 and Appendix A. This dataset covers more realistic conversational and workflow-oriented scenarios, including explanations, procedures, and stories, where users interrupt to query or resume previously spoken content. All test samples were newly generated and excluded from training.
As shown below, our 3-channel Self-Listening model achieves 56.67% accuracy, outperforming the strongest open-source baseline (21.67%) and our 2-channel FD model (6.67%) by 35.0 and 50.0 percentage points, respectively. These results provide further evidence of the effectiveness of Self-Listening in more complex semantic anchoring scenarios beyond deterministic sequences.
Table 1. Performance comparison on the Semantic Anchor-Interruption Test Set (accuracy, %, ↑)