acceptodds
Under review as a conference paper at ICLR 2027

What Did I Actually Say? Self-Listening for Full-Duplex Speech Models

Abstract

Full-duplex spoken language models can listen and speak simultaneously, enabling them to handle interruptions and backchannels in human conversation. However, text generation, speech synthesis, and audio playback proceed asynchronously. As a result, what a model believes it has said may not match what has actually been played to the user. We refer to the problem of recovering from an interruption while remaining aware of the assistant's realized speech as anchor interruption. To address this problem, we propose self-listening, a full-duplex modeling approach that interleaves user speech, assistant text, and the assistant's played speech. By feeding the realized speech output back to the model as an input stream, self-listening grounds interruption recovery in what the user has actually heard. We further introduce AnchorSpeech, which contains both training data and a controlled evaluation set for fine-grained anchoring. AnchorSpeech evaluates whether a model can accurately identify the spoken boundary reached before an interruption. Experiments show that, compared with a full-duplex baseline, models equipped with self-listening achieve better anchoring performance and produce more coherent responses after interruption.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Full-duplex Speech Models, Spoken Language Models, Spoken Dialogue

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_12saOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper handles whether full-duplex speech models can accurately track how far they have spoken when interrupted by a user. The authors introduce AnchorSpeech, a benchmark and training dataset for this capability, and propose Self-Listening, which feeds the model’s played speech back into its input. Experiments show that existing models struggle with this task, while the proposed model achieves substantially higher anchoring accuracy.

Critical strengths
  1. The paper identifies an underexplored problem in full-duplex speech models and introduces a benchmark and a modeling approach to address it. The proposed model shows substantial improvements on the benchmark.

  2. The authors also evaluate their model on Full-Duplex-Bench v1.5 to assess its broader full-duplex conversational capabilities.

Critical weaknesses
  1. The claims around Self-Listening appear overstated. Moshi, PersonaPlex, SALMONN-omni, MiniCPM-o 4.5, and Raon-SpeechChat already incorporate representations of their own outputs into subsequent inputs while processing user speech. The claimed distinction is therefore playback alignment (mentioned in related work), not self-listening itself. The paper does not sufficiently establish what additional information this feedback provides over existing output histories or why it requires the proposed architecture.

  2. The implementation of playback alignment is unclear. Feeding back speech when it is actually played requires synchronization with user-side playback, accounting for generation latency, network delays, and buffering. These details are missing. If the model simply receives generated speech intended for playback, its distinction from existing approaches is unclear.

  3. The proposed model is trained and evaluated on similarly constructed data. While valid, this setting does not establish a structural advantage or generalization to realistic interactions and different data distributions.

  4. Existing full-duplex models should be fine-tuned on the same data, including AnchorSpeech and InstructS2S, and compared under matched conditions to establish the benefit of Self-Listening. This comparison is essential to determine whether the gains come from the proposed feedback mechanism or task-specific training.

  5. The evaluation lacks broader measures of conversational dynamics, semantic capabilities, and dialogue quality, such as the full Full-Duplex-Bench v1, VoiceBench, and human evaluation. Results on a subset from Full-Duplex-Bench v1.5 alone are insufficient to establish the model’s overall full-duplex conversational capabilities.

Overall, the paper raises a relevant question, but the current evidence does not sufficiently establish the novelty, necessity, or generalizability of the proposed approach.

Questions for authors
  1. Could the authors clarify the precise distinction between Self-Listening and existing approaches? Despite repeated reading, I find the current explanation insufficient to identify the substantive architectural difference.

  2. Could the authors demonstrate the benefit of this distinction by additionally training existing full-duplex models on InstructS2S-200K and AnchorSpeech and comparing their performance under matched conditions?

  3. Could the authors evaluate the model on the full Full-Duplex-Bench v1 and semantic benchmarks such as OpenAudioBench or VoiceBench to establish its broader capabilities as a full-duplex speech model?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 12sa (1)

We thank the reviewer for recognizing the importance of the anchoring problem and for the valuable feedback and constructive suggestions. Below, we address the reviewer’s main concerns.

W1 & Q1: Novelty and Architectural Distinction

We thank the reviewer for the insightful discussion regarding the distinction between Self-Listening and existing full-duplex models. We agree that conditioning subsequent predictions on previously generated speech has already been adopted by models such as Moshi and MiniCPM-o. We would like to clarify that the key distinction lies in what information the feedback represents and what capabilities the model can acquire from it.

Existing models such as Moshi incorporate previously generated speech tokens into their autoregressive context to support continuous speech generation. However, their audio feedback reflects generation progress rather than actual playback progress. Due to speech synthesis and playback delays, the model's autoregressive context may already contain speech that has not yet been delivered to the user. Consequently, when an interruption occurs, the model cannot reliably determine the actual spoken boundary from its generation history alone, making it difficult to accurately track how far it has spoken. This limitation is also reflected in our experiments, where Moshi and MiniCPM-o 4.5 achieve anchoring accuracies of only 6.7% and 24.7%, respectively, on AnchorSpeech-test.

In contrast, Self-Listening continuously feeds the model's actual speaking progress back into the model through a dedicated feedback channel, independently of its internal generation history. By explicitly modeling the generated content and the actually played speech as separate streams, the model can distinguish between what it has generated and what the user has actually heard, even when speech generation and playback are asynchronous. Through targeted anchor-interruption training, the model learns to associate its generation history with the realized speech progress, allowing it to accurately identify how far its speech has reached the user and recover from the correct position when interrupted.

W2: Playback Alignment Implementation

We thank the reviewer for raising this concern. Our implementation accounts for speech generation latency and audio buffering. Specifically, since our model is deployed locally, we can directly track the audio player's playback position. Generated speech is first placed in the playback buffer, and only audio segments that have actually been played are fed back into the Self-Listening channel. Unplayed audio is excluded, ensuring that the feedback remains aligned with actual playback progress, even when generation and playback are asynchronous.

Since our original experiments use local deployment, they do not involve network transmission delays. To address this concern, we conducted an additional robustness ablation by introducing random feedback delays of 0–400 ms into the Self-Listening channel. The anchoring accuracy remains between 70% and 75%, comparable to approximately 74% in the original setting. These results suggest that Self-Listening is robust to moderate feedback transmission delays and maintains stable anchoring performance under the tested conditions.

W3: Generalization to Realistic Conversational Scenarios

To the best of our knowledge, there is currently no established benchmark explicitly evaluating whether a model can accurately track its own speaking progress during interruptions. As the first work to systematically investigate this problem, we had to construct both the training and evaluation datasets ourselves, making some similarity in their data distributions difficult to avoid. We acknowledge this limitation and hope that future work will further investigate anchoring in diverse real-world conversational settings.

Nevertheless, to better evaluate generalization to realistic scenarios, we additionally constructed a new held-out Semantic Anchor-Interruption Test Set, following the procedure in Section 4.2 and Appendix A. This dataset covers more realistic conversational and workflow-oriented scenarios, including explanations, procedures, and stories, where users interrupt to query or resume previously spoken content. All test samples were newly generated and excluded from training.

As shown below, our 3-channel Self-Listening model achieves 56.67% accuracy, outperforming the strongest open-source baseline (21.67%) and our 2-channel FD model (6.67%) by 35.0 and 50.0 percentage points, respectively. These results provide further evidence of the effectiveness of Self-Listening in more complex semantic anchoring scenarios beyond deterministic sequences.

Table 1. Performance comparison on the Semantic Anchor-Interruption Test Set (accuracy, %, ↑)

ModelMoshiFreeze-OmniPersonaPlex2-ch FD Model3-ch Self-Listening
Acc (%, ↑)10.0018.3321.676.6756.67
AuthorsOpenReview ↗

Response to Reviewer 12sa (2)

W4 & Q2: Fair Comparison and Controlled Ablation

To isolate the contribution of the Self-Listening mechanism from that of task-specific training data, we have already conducted a controlled ablation study comparing 2-channel and 3-channel models in our original paper. Both models share the same Qwen2.5-Omni backbone and training data, including AnchorSpeech, with the primary difference being the presence of the Self-Listening channel. Despite being trained on the same data, the 2-channel model achieves only 7.8% anchoring accuracy on AnchorSpeech-test, whereas the 3-channel Self-Listening model achieves 73.0%. These results demonstrate that the improvement cannot be attributed solely to task-specific training data, highlighting the critical role of explicit playback-grounded feedback in achieving reliable anchoring.

We agree that fine-tuning existing full-duplex models on the same data would provide additional insights. However, retraining all these models is challenging given the computational resources and training costs involved. Moreover, these models differ substantially in their pretraining data, architectures, speech representations, and training strategies, making it difficult to ensure a strictly fair comparison even with identical fine-tuning data. Therefore, we adopted a controlled ablation with the same backbone and training configuration to more directly isolate the contribution of Self-Listening. We leave broader cross-architecture comparisons under unified training conditions for future work.

W5 & Q3: Broader Conversational Capabilities

(1) Additional evaluation on Full-Duplex-Bench v1.

Following the reviewer's suggestion, we further evaluated our model on the complete Full-Duplex-Bench v1, covering Pause Handling, Smooth Turn-taking, Backchannel, and User Interruption. As shown in Table 1, our model achieves the best performance on several Pause Handling and Backchannel metrics, while remaining competitive on Smooth Turn-taking. Since our training data primarily focuses on user interruptions and anchoring, without specifically incorporating False Handling and Model Interruption scenarios, some performance gaps in general interaction metrics are understandable. Overall, these results demonstrate that our model maintains competitive full-duplex conversational capabilities while acquiring the additional anchoring capability.

Table 2. Evaluation on Full-Duplex-Bench v1.

TaskDatasetMetricPersonaPlexdGSLMMoshiFreeze-Omni3-ch Self-Listening
Pause HandlingSynthetic (137)TOR ↓0.5840.9340.9850.6420.5474
Pause HandlingCandor (216)TOR ↓0.6620.9350.9800.4810.5926
Smooth Turn-takingCandor (119)TOR ↑0.9920.9750.9410.3360.9664
Smooth Turn-takingCandor (119)Latency (s) ↓0.0700.3520.2650.9530.2642
BackchannelICC (55)TOR ↓0.3270.6911.0000.6360.1455
BackchannelICC (55)Frequency (/s) ↑0.0250.0150.0010.0010.0250
BackchannelICC (55)JSD ↓0.6490.9340.9570.9970.8901
User InterruptionSynthetic (200)TOR ↑1.0000.9171.0000.8670.9750
User InterruptionSynthetic (200)GPT Score (0–5) ↑4.2100.2010.7653.6152.5436
User InterruptionSynthetic (200)Latency (s) ↓0.4002.5310.2571.4090.3138

(2) Additional evaluation of semantic capabilities.

We further evaluated our model on three QA benchmarks: Llama Questions, WebQuestions, and TriviaQA. As shown in Table 2, our 3-channel Self-Listening model achieves accuracies of 69.0% and 37.9% on Llama Questions and TriviaQA, respectively, outperforming all evaluated full-duplex baselines, while remaining competitive on WebQuestions. These results indicate that our model retains strong general knowledge and question-answering capabilities while improving anchoring performance.

Table 3. QA performance compared with representative full-duplex models (accuracy, %, ↑).

DatasetMoshiFreeze-OmniPersonaPlex3-ch Self-Listening
Llama Questions54.556.261.769.0
WebQuestions22.127.922.523.4
TriviaQA16.728.522.837.9
AuthorsOpenReview ↗

Kindly Reminder

Thank you again for your time and thoughtful review. With the discussion period closing in one day, we would appreciate hearing whether our responses have addressed your concerns and whether any outstanding issues remain. If your concerns have been resolved, we would be grateful if you would consider updating your rating accordingly.

Reviewer_Gf6NOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes a "Self-Listening" mechanism to address the "anchor interruption" problem in full-duplex speech models caused by the asynchronous text generation and audio playback processes. The method feeds the played model speech back as an additional input channel, allowing the model to ground its interruption recovery on what the user has actually heard. Additionally, the paper introduces a synthetic dataset, AnchorSpeech, to train and evaluate this fine-grained progress tracking capability. Experiments show that this mechanism effectively improves the anchoring accuracy of the system during interruption recovery.

Critical strengths
  • Clear Problem Definition and Benchmarking: The paper successfully identifies and quantifies the "Anchoring Gap" present in existing full-duplex systems (such as GPT-Realtime-2.1 and Doubao-Realtime), clearly defining the scope of the problem through a probing experiment.
  • Substantial Improvement in Anchoring Accuracy: On the AnchorSpeech-test, the model equipped with the three-channel self-listening mechanism achieved an anchoring accuracy of 73.0%, substantially outperforming the evaluated closed-source (max 47.2%) and open-source baselines. This effectively validates the technical feasibility of the method in addressing the "self-listening blind spot".
Critical weaknesses
  • Trade-offs in Interaction Performance: The channel ablation study (Table 3) shows that after introducing the self-listening channel, although the anchoring accuracy rises sharply, it is accompanied by increased latency and a decreased response rate. Specifically, the STOP latency increases from 0.43 to 0.81 seconds, and the RESP latency increases from 0.57 to 0.66 seconds. Simultaneously, in the Interruption setting of Full-Duplex-Bench v1.5, the RESPOND rate drops from 0.83 to 0.60. Overcoming this challenge to maintain agile responsiveness to interruptions while preserving anchoring precision is an important consideration for practical application.
  • Limited Evaluation Scenarios: The introduction mentions that this capability is critical for "structured workflows, such as navigation, troubleshooting, interactive tutoring". However, the AnchorSpeech test set used for quantitative evaluation primarily focuses on relatively fixed and regular sequences like counting, countdowns, and reciting alphabets. There is a noticeable gap between the evaluation cases and the complex application scenarios described in the introduction.
  • Missing Evaluation on Semantic Anchors: The paper mentions the construction of "semantic anchor-interruption" data covering open-ended dialogues (e.g., code, formulas, stories). However, this data is currently only used as a supplementary training source, and no independent test set or quantitative evaluation results for semantic anchors are reported. This limits the readers' comprehensive understanding of the model's true performance in handling interruption recovery during unstructured dialogues.
Questions for authors
  1. Underlying Cause of Responsiveness Degradation: In Table 3, both the 2-ch and 3-ch models share the same backbone, training data, and 400 ms <overlap> reaction window. Why does introducing the sts_t stream cause the Interruption RESPOND rate to drop from 0.83 to 0.60 and STOP latency to nearly double? Is this primarily caused by acoustic interference/confusion between the user speech stream (utu_t) and played model speech stream (sts_t) within the shared attention space, or by the 1.5x increase in sequence token density per logical step?
  2. Comparison with System-Level Playback Tracking: Since the model generates text before sending it to the TTS module, why is acoustic self-listening necessary compared to a system-level synchronization mechanism (e.g., having the audio player report the last played word/token index via TTS alignment and rolling back the LLM's KV cache or text context accordingly)? Could the authors clarify the advantages of acoustic re-encoding over deterministic playback-pointer tracking?
  3. Evaluation on Open-Ended Semantic Anchoring: Given that semantic anchor-interruption data (covering stories, procedures, and explanations) was already constructed for Stage-2 training (Appendix A & C.2), why was an evaluation split not held out to test semantic anchoring? Can the authors provide quantitative results on a held-out test set of semantic anchor interruptions to demonstrate that the method generalizes beyond deterministic sequences like counting and spelling?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Gf6N (1)

We sincerely thank the reviewer for recognizing the significance of the underexplored anchoring problem, the effectiveness of our proposed approach, and our evaluation of broader full-duplex conversational capabilities. We also appreciate the valuable feedback and constructive suggestions. Below, we address the reviewer's main concerns.

W1: Trade-offs in Interaction Performance. & Q1: Underlying Cause of Responsiveness Degradation

(1) Responsiveness Degradation. To further investigate the decrease in interruption responsiveness, we conducted two controlled ablation experiments. The results suggest that the degradation is primarily associated with interference among training objectives, rather than increased token density or acoustic interference between the user-speech and model-speech streams.

First, we compared the 2-channel model with a 3-channel variant whose self-listening channel contains only <silence> tokens. As shown in Table 1, their full-duplex performance remains comparable (RESPOND: 0.81 vs. 0.78; RESUME: 0.88 vs. 0.88), indicating that the additional sequence tokens alone do not substantially impair responsiveness.

Table 1.Effect of the additional channel with silence-only inputs on Full-Duplex-Bench v1.5.

ScenarioMetric2-ch FD3-ch (Silence Only)
User InterruptionRespond ↑0.810.78
Stop Latency (s) ↓1.51311.2566
Response Latency (s) ↓0.65510.7003
BackchannelResume ↑0.880.88
Stop Latency (s) ↑0.71550.7030
Response Latency (s) ↓0.70110.7416

Second, we compared the 2-channel and 3-channel models trained exclusively on general full-duplex speech data, without anchor-interruption training. Here, the 3-channel model receives actual model-speech feedback through its self-listening channel. As shown in Table 2, both models achieve comparable responsiveness (RESPOND: 0.57 vs. 0.59; RESUME: 0.98 vs. 0.99), suggesting that acoustic interference between the two speech streams is not the primary cause. Therefore, these results suggest that the larger responsiveness gap observed after anchor training is more likely associated with competing training objectives, as introducing additional tasks can lead to partial forgetting of previously learned capabilities, a commonly observed phenomenon in multi-task learning.

Table 2. Full-Duplex-Bench v1.5 results without anchor-interruption training.

ScenarioMetric2-ch FD3-ch Self-Listening
User InterruptionRespond ↑0.590.57
Stop Latency (s) ↓1.35681.1797
Response Latency (s) ↓0.71180.7993
BackchannelResume ↑0.990.98
Stop Latency (s) ↑0.72350.7185
Response Latency (s) ↓0.71640.6550

(2) Latency Analysis. The reported latency differences are influenced by the evaluation protocol of Full-Duplex-Bench v1.5, it includes both successful and unsuccessful interruptions in latency evaluation. Since unsuccessful interruptions tend to incur higher measured latency, a lower response rate can lead to higher aggregate latency. As shown in Tables 1 and 2, when the 2-channel and 3-channel models achieve comparable response rates, their latencies also remain similar. These results suggest that our self-listening mechanism does not introduce substantial additional latency overhead.

W2: Limited Evaluation Scenarios. & W3: Missing Evaluation on Semantic Anchors & Q3: Evaluation on Open-Ended Semantic Anchoring.

We appreciate the reviewer's concern regarding the limited evaluation scenarios. The original AnchorSpeech-test focuses on deterministic sequences to enable a more precise and objective evaluation of differences in models' anchoring capabilities, with clearly defined ground truth. To further assess scenarios beyond these structured sequences, we constructed a new held-out Semantic Anchor-Interruption Test Set following the data construction procedure in Section 4.2 and Appendix A. This test set covers more realistic conversational scenarios, including explanations, procedures, and stories, where users interrupt to query or resume previously spoken content. All test samples were newly generated and excluded from training. As shown below, our 3-channel Self-Listening model achieves 56.67% accuracy, outperforming the strongest open-source baseline (21.67%) and our 2-channel FD model (6.67%) by 35.0 and 50.0 percentage points respectively, demonstrating improved semantic anchoring in more complex conversational scenarios.

Table 3. Semantic Anchor-Interruption Test Set.

ModelMoshiFreeze-OmniPersonaPlex2-ch FD Model3-ch Self-Listening
Acc(%,↑)10.0018.3321.676.6756.67
AuthorsOpenReview ↗

Response to Reviewer Gf6N (2)

Q2: Comparison with System-Level Playback Tracking

We appreciate the reviewer's constructive suggestion. Our goal is to explore a learning-based, end-to-end approach that enables models to track their own realized speech without relying on manually designed synchronization mechanisms. While system-level playback tracking is a reasonable alternative, it has two potential limitations. First, TTS systems may not directly provide word-level playback alignment, requiring additional ASR-based alignment that can introduce latency. Second, truncating unplayed text from the model's context may discard useful generated information, potentially affecting its reasoning capabilities, especially in future reasoning-intensive speech models.

Following the reviewer's suggestion, we evaluated two inference-only approaches on our 2-channel FD model without retraining: (1) Text Feedback, which uses ASR to identify the played speech prefix and inserts a model-only text prompt into the context to indicate how far the previous response has been played; (2) Text Truncation, which removes unplayed text from the model's context based on the estimated playback position.

Table 4. Comparison with system-level playback tracking on AnchorSpeech-test (accuracy, %, ↑).

Method2-ch FDText FeedbackText Truncation3-ch Self-Listening
Anchoring Accuracy13.5021.1765.0673.00

Both system-level approaches improve anchoring over the original 2-channel model, but remain below our Self-Listening model (73.00%). Moreover, the additional ASR processing introduces approximately 2–10 seconds of latency in our implementation (we did not optimize the ASR pipeline for streaming due to rebuttal time constraints). We acknowledge that optimized system-level approaches may further improve anchoring performance. Nevertheless, our goal is to investigate how models can learn to track their own realized speech, laying the foundation for more autonomous and robust full-duplex interaction without relying on handcrafted synchronization mechanisms.

AuthorsOpenReview ↗

Kindly Reminder

Thank you again for your time and thoughtful review. With the discussion period closing in one day, we would appreciate hearing whether our responses have addressed your concerns and whether any outstanding issues remain. If your concerns have been resolved, we would be grateful if you would consider updating your rating accordingly.

Reviewer_Gf6NOpenReview ↗

Official Comment by Reviewer Gf6N

Dear authors,

Thank you for the detailed response and the substantial additional experiments. Some of my concerns have been addressed, including the cause analysis of the responsiveness degradation and the new semantic anchoring evaluation. These additions strengthen the empirical support for the paper's claims, and I will raise my soundness score accordingly.

However, my main concern about the significance of the contribution remains. In particular, the new comparison shows that a training-free Text Truncation baseline already reaches 65.06% vs. 73.0% for Self-Listening, suggesting that much of the anchoring problem can be handled by system-level engineering, while the responsiveness trade-off is diagnosed but not yet mitigated. The main contribution thus seems to lie in problem identification and targeted data curation, which I find valuable but somewhat limited in architectural novelty. Therefore I intend to maintain my current overall rating.

Reviewer_qYPdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Self-Listening, a full-duplex modeling approach that incorporates a model's own generated speech into its input stream. The authors first conduct diagnostic experiments demonstrating that even commercial conversational AI systems cannot consistently recall their own generated speech, highlighting the importance of feeding generated speech back into the model. They then develop a full-duplex conversational model equipped with the proposed self-listening mechanism and evaluate its performance across diverse benchmarks, including the newly introduced AnchorSpeech, which is specifically designed to assess whether a model can recognize and reason about what it has already said.

Critical strengths
  1. Drawing inspiration from the human auditory self-monitoring mechanism provides an interesting motivation for the proposed self-listening approach. The diagnostic pilot study investigating whether existing conversational AI systems can accurately recall their own utterances is also insightful.

  2. The proposed AnchorSpeech benchmark addresses an underexplored aspect of conversational AI: whether a model can accurately recognize and reason about its own previously generated speech.

Critical weaknesses

The primary concern is the limited architectural novelty of the proposed approach. Similar self-listening mechanisms have already been implemented in Moshi [1] and SALMONN-omni [2]. Therefore, the proposed method cannot be regarded as a fundamentally novel architectural design. Rather, that existing full-duplex architectures already possess the capacity to address this problem, but previous studies have not explicitly curated task-specific data to develop and evaluate this capability. Consequently, the main contributions of this work appear to lie in identifying an underexplored problem and curating targeted data to address it, rather than introducing a novel modeling architecture.

[1] Défossez, Alexandre, et al. "Moshi: a speech-text foundation model for real-time dialogue." arXiv preprint arXiv:2410.00037 (2024). [2] Yu, Wenyi, et al. "Salmonn-omni: A standalone speech llm without codec injection for full-duplex conversation." Advances in Neural Information Processing Systems 38 (2026): 24617-24643.

Questions for authors
  1. The current evaluation primarily focuses on full-duplex interaction capabilities, while the quality of generated textual responses and the model's general language capabilities remain unexplored. Could the authors evaluate the proposed model on established question-answering benchmarks, such as WebQuestions and TriviaQA, to assess its general knowledge and reasoning capabilities? Furthermore, an ablation study would be valuable to investigate whether incorporating the model's own generated speech into the input stream adversely affects its language capabilities.

  2. The AnchorSpeech-test set consists entirely of synthetically generated samples, raising concerns about how faithfully it reflects real-world conversational scenarios. Could the authors incorporate real-world conversational data into the benchmark and provide stronger evidence of the proposed method's effectiveness in practical settings?

4 comments
AuthorsOpenReview ↗

Response to Reviewer qYPd (1)

We thank the reviewer for recognizing the motivation and research problem addressed in our work, as well as for the valuable feedback and constructive suggestions. Below, we address the reviewer's main concerns.

W1: Limited architectural novelty

We thank the reviewer for the insightful discussion regarding the distinction between Self-Listening and existing full-duplex models. We agree that conditioning subsequent predictions on previously generated speech has already been adopted by models such as Moshi and MiniCPM-o. We would like to clarify that the key distinction lies in what information the feedback represents and what capabilities the model can acquire from it.

Existing models such as Moshi incorporate previously generated speech tokens into their autoregressive context to support continuous speech generation. However, their audio feedback reflects generation progress rather than actual playback progress. Due to speech synthesis and playback delays, the model's autoregressive context may already contain speech that has not yet been delivered to the user. Consequently, when an interruption occurs, the model cannot reliably determine the actual spoken boundary from its generation history alone, making it difficult to accurately track how far it has spoken. This limitation is also reflected in our experiments, where Moshi and MiniCPM-o 4.5 achieve anchoring accuracies of only 6.7% and 24.7%, respectively, on AnchorSpeech-test.

In contrast, Self-Listening continuously feeds the model's actual speaking progress back into the model through a dedicated feedback channel, independently of its internal generation history. By explicitly modeling the generated content and the actually played speech as separate streams, the model can distinguish between what it has generated and what the user has actually heard, even when speech generation and playback are asynchronous. Through targeted anchor-interruption training, the model learns to associate its generation history with the realized speech progress, allowing it to accurately identify how far its speech has reached the user and recover from the correct position when interrupted.

Q1: The quality of generated textual responses.

To evaluate response quality, we conducted additional experiments on three QA benchmarks and three full-duplex models under the same speech-to-speech evaluation protocol. As shown in Table 1, our 3-channel Self-Listening model outperforms all baselines on Llama Questions and TriviaQA while remaining competitive on WebQuestions. We further conducted an ablation study to investigate whether incorporating the model's own generated speech into the input stream affects its language capabilities. As shown in Table 2, compared with the original half-duplex Qwen2.5-Omni and our 2-channel full-duplex model, the 3-channel Self-Listening model exhibits a moderate decline in QA performance, indicating a trade-off between the additional self-listening capability and response quality.

Table 1. Comparison with representative full-duplex models (accuracy, %, ↑).

DatasetMoshiFreeze-OmniPersonaPlex3-ch Self-Listening
Llama Questions54.556.261.769.0
WebQuestions22.127.922.523.4
TriviaQA16.728.522.837.9

Table 2. QA performance across different duplex configurations (accuracy, %, ↑).

DatasetQwen2.5-Omni2-ch FD Model3-ch Self-Listening
Llama Questions75.074.369.0
WebQuestions29.925.823.4
TriviaQA52.844.237.9

It is worth noting that enhancing real-time conversational interactivity can come at the cost of semantic understanding and reasoning capabilities, posing a common challenge for full-duplex speech models. For instance, Moshi [1] reports an MMLU decline from 54.3% to 49.7% following multimodal and multi-stream training, while D2 [2] observes a VoiceBench drop from 86.8 to 70.1 when adapting Qwen3-Omni from half-duplex to full-duplex. Lychee-FD [3] further identifies gradient conflicts between acoustic and semantic modeling as a potential cause of knowledge degradation. More broadly, similar degradation has been observed in multimodal adaptation: WINGS [4] reports a 9.7-point MMLU decrease after vision-language training, while Xiang et al. [5] observe an 18.1-point drop in text-based QA after speech-text alignment training. Consistent with these findings, our results reflect the challenge of preserving QA capabilities when extending the model with full-duplex and Self-Listening functionality, while still maintaining competitive performance against existing full-duplex models.

AuthorsOpenReview ↗

Response to Reviewer qYPd (2)

Q2: Evaluation in real-world conversational scenarios.

We appreciate the reviewer's concern regarding real-world applicability. To further evaluate our method, we constructed a new held-out Semantic Anchor-Interruption Test Set, following the data construction procedure in Section 4.2. Unlike the sequence-based AnchorSpeech-test, this dataset covers more realistic conversational scenarios, such as explanations, procedures, and stories, where users interrupt to query or resume previously spoken content. All test samples were newly generated and excluded from training. As shown below, our 3-channel Self-Listening model achieves 56.67% accuracy, outperforming the strongest open-source baseline (21.67%) and our 2-channel FD model (6.67%) by 35.0 and 50.0 percentage points, respectively, demonstrating improved semantic anchoring in more complex conversational scenarios.

Table3. Semantic Anchor-Interruption Test Set.

ModelMoshiFreeze-OmniPersonaPlex2-ch FD Model3-ch Self-Listening
Acc(%,↑)10.0018.3321.676.6756.67

[1] Défossez et al., Moshi: A Speech-Text Foundation Model for Real-Time Dialogue, arXiv, 2024.

[2] Chang et al., D2: High-Capability Full-Duplex Spoken Language Models from Half-Duplex Backbones, arXiv, 2026.

[3] Liu et al., Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs, ACL, 2026.

[4] Zhang et al., WINGS: Learning Multimodal LLMs without Text-only Forgetting, NeurIPS, 2024.

[5] Xiang et al., Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Models, EMNLP, 2025.

AuthorsOpenReview ↗

Kindly Reminder

Thank you again for your time and thoughtful review. With the discussion period closing in one day, we would appreciate hearing whether our responses have addressed your concerns and whether any outstanding issues remain. If your concerns have been resolved, we would be grateful if you would consider updating your rating accordingly.

I thank the authors for their rebuttal. Although some of my concerns have been addressed, I still believe that the proposed method primarily represents a modification to the data organization rather than a substantial architectural innovation. In my view, the contribution would be more convincing if the paper shifted its emphasis from developing a new model to demonstrating the effectiveness and generalizability of the proposed data arrangement strategy. (which also means the paper should be reorganized to a new paper.) Specifically, applying this technique to existing full-duplex models, such as Moshi or MiniCPM-o, and evaluating whether it improves their ability to recognize and recall their own generated speech would provide stronger evidence of its value. Therefore, while I appreciate the authors' efforts to address my concerns, I will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.