acceptodds
Under review as a conference paper at ICLR 2027

Bridge-Duplex: Bridging Semantic and Generative Streams for Full-Duplex Speech-to-Speech Dialogue

Abstract

Full-duplex speech-to-speech (S2S) dialogue requires dialogue-level semantic processing to remain coordinated with continuous frame-wise generation as conversational evidence evolves. Existing systems increasingly make semantic computation explicit, but how semantic state should persist, inform ongoing generation, and be updated throughout dialogue remains underexplored. We propose Bridge-Duplex, a persistent semantic–generative dual-stream framework for full-duplex S2S dialogue. Bridge attention connects the two streams on a shared causal timeline, exposing persistent semantic memory to ongoing generation. Dialogue next-latent prediction further regularizes semantic state evolution by predicting each subsequent semantic state from the current user–system state pair and next-token evidence during training. Bridge-Duplex improves multi-turn dialogue quality and context consistency over the underlying full-duplex models. Selective bridge attention preserves most of the multi-turn semantic performance of dense bridge attention while improving inference efficiency through selective semantic memory access. Audio samples are available at https://anonymous77077.github.io/Bridge-Duplex_Demo/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
full-duplex speech-to-speech dialogue, speech language models, spoken dialogue systems, streaming speech generation, semantic memory, sparse attention

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kUqXOpenReview ↗
Rating
3
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Bridge-Duplex, a dual-stream full-duplex S2S framework that couples a persistent semantic stream with a frame-wise generative stream via selective bridge attention and training-time next-latent prediction, significantly improving multi-turn semantic consistency efficiently.

Critical strengths

The motivation is clear and well supported by the preliminary K/V pruning experiment, which shows that the semantic and generative streams have different context dependencies. The experimental evaluation is quite thorough, testing the approach across two different backbones (F-Actor and a LoRA-adapted Moshi) and including a 3B baseline to account for model size. The performance gains on MTR-DB and Behavior-SD are solid, especially regarding dialogue quality and multi-turn consistency.

Critical weaknesses

While the framework excels in full-duplex conversational interaction, the evaluation is largely confined to dialogue chit-chat datasets. It lacks validation on broader, general-purpose speech LLM benchmarks to demonstrate whether the base model's general reasoning and factual knowledge capabilities are well-preserved under this full-duplex formulation.

Questions for authors

Overall, I have no other major concerns with the paper. My only question is whether the authors could consider evaluating on general speech language model benchmarks such as VoiceBench as well as speech understanding benchmarks such as MMAU. The method clearly performs well in full duplex settings, but it would be good to verify that performance on standard reasoning and general speech understanding does not degrade.

5 comments
AuthorsOpenReview ↗

Author Response 1: Summary

We thank the reviewer for the positive assessment and valuable suggestion. To address the concern about general speech capabilities beyond conversational dialogue, we provide additional evidence on the following aspect:

  1. General speech capabilities: We extend our evaluation to Big Bench Audio, selected VoiceBench tasks, and an adapted MMAU-Speech setting. Bridge-Duplex consistently outperforms the corresponding conversation-adapted backbones across the reported tasks, providing further evidence of its benefits for general speech understanding and reasoning alongside multi-turn dialogue.

We provide the evaluation details and quantitative results below.

AuthorsOpenReview ↗

Author Response 2: Detailed Response

Comment 1 (W1, Q1): General Speech Evaluation beyond Dialogue Benchmarks

We have added evaluations of spoken reasoning with Big Bench Audio (BBA) [R1], knowledge-based and open-ended question answering with selected VoiceBench tasks [R2], and speech understanding with MMAU-Speech [R3]. Table R1 summarizes these complementary evaluations.

We include both pretrained Moshi and a Moshi baseline fine-tuned on conversational speech data to distinguish pretrained capability retention from performance following conversational adaptation. Text-only models serve as reference points rather than modality-matched baselines. For MMAU-Speech, we synthesize textual prompts using TTS for speech-input models, while text-only LLMs receive ASR transcriptions of the audio alongside the original text prompts. Unlike our dialogue benchmarks, MMAU-Speech focuses on fine-grained speech understanding and reasoning tasks that are less aligned with our conversational speech-focused adaptation.

Table R1. Results on BBA, selected VoiceBench tasks, and MMAU-Speech. Higher scores are better within each column. BBA categories, OpenBookQA, and MMAU-Speech report accuracy (%); AlpacaEval and CommonEval report LLM-judged response-quality scores on a 1–5 scale.

ModelInputFormal Fallacies (BBA)Navigate (BBA)Object Counting (BBA)Web of Lies (BBA)OpenBookQA (VoiceBench)AlpacaEval (VoiceBench)CommonEval (VoiceBench)MMAU-Speech (MMAU)
Helium-6BText25.6 %51.6 %28.4 %15.2 %41.10 %3.211.7426.7 %
Llama-3.2-1B-InstructText26.4 %30.0 %35.6 %20.8 %53.19 %3.912.7932.7 %
Moshi (Pretrained)Audio40.0 %12.0 %15.2 %28.4 %17.36 %3.071.340.00 %
Moshi (Conversation-adapted)Audio0.0 %3.6 %0.8 %0.4 %5.49 %2.021.093.60 %
Selective Bridge-Duplex (Moshi)Audio18.4 %19.2 %15.6 %4.0 %18.2 %2.061.746.91 %
F-Actor (Conversation-adapted)Audio0.0 %2.4 %0.0 %1.6 %0.00 %1.391.011.50 %
Selective Bridge-Duplex (F-Actor)Audio2.0 %12.4 %0.8 %12.4 %12.8 %2.811.853.60 %

As shown in Table R1, Bridge-Duplex achieves higher scores than the corresponding conversation-adapted baselines on the reported tasks across both backbones. However, conversational fine-tuning of the Moshi baseline is also associated with reduced general reasoning and question-answering performance on several evaluated tasks. To contextualize this pattern, Table R2 summarizes the adaptation data, multi-turn semantic performance, and average BBA accuracy of the Moshi-based models.

Table R2. Training-data composition and conversational and spoken-reasoning performance of Moshi-based models.

ModelTask-specific adaptation dataDQ / CC ↑ (Behavior-SD)DQ / CC ↑ (MTR-DB)BBA Avg. Accuracy ↑
Moshi (Pretrained)— (no additional adaptation)2.91 / 3.491.99 / 2.6423.90 %
Moshi (Conversation-adapted)Behavior-SD (conversational speech)3.72 / 4.402.07 / 2.601.20 %
Bridge-Duplex (Moshi backbone)Behavior-SD (conversational speech)3.82 / 4.593.01 / 4.3414.30 %

Taken together, Tables R1 and R2 show that conversational adaptation improves the Moshi baseline’s conversational performance while reducing its performance on several general reasoning and question-answering tasks. Under the same adaptation data, Bridge-Duplex achieves stronger results than this adapted baseline in both multi-turn dialogue and the reported general-capability evaluations. However, the remaining gap to pretrained Moshi on BBA indicates that these improvements do not imply uniform preservation of pretrained capabilities. These findings motivate future work on conversational adaptation strategies that better retain pretrained reasoning and question-answering capabilities while improving multi-turn dialogue.

[R1] M. Hill-Smith and G. Cameron, “Evaluating Audio Reasoning with Big Bench Audio,” Artificial Analysis Blog, 2024. (https://artificialanalysis.ai/methodology/speech-to-speech-benchmarking#speech-reasoning)

[R2] Chen, Yiming, et al. "Voicebench: Benchmarking llm-based voice assistants." Transactions of the Association for Computational Linguistics 14 (2026): 378-398.

[R3] Sakshi, Sakshi, et al. "Mmau: A massive multi-task audio understanding and reasoning benchmark." International Conference on Learning Representations. Vol. 2025. 2025.

AuthorsOpenReview ↗

We would like to follow up on our rebuttal

Dear Reviewer kUqX,

We sincerely thank you once again for your valuable time and effort in reviewing our manuscript.

We deeply appreciate your insightful comments and we have worked to address your concerns through additional experiments and clarifications. Following your suggestion, we have provided additional evaluations on Big Bench Audio, selected VoiceBench tasks, and an adapted MMAU-Speech setting in our rebuttal.

We hope these comments and improvements can be taken into account when deciding on the final assessment for paper. We remain open to further discussion and are happy to provide additional clarifications if needed.

Thank you once again for your constructive feedback and the opportunity to improve our work.

sincerely,

The Authors.

Reviewer_xrUeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Bridge-Duplex, a dual-stream framework for full-duplex speech-to-speech dialogue. A persistent semantic stream processes role-separated dialogue text, while a generative stream produces frame-wise audio and text. Causal bridge attention exposes semantic memory to ongoing generation, and a selective variant retrieves relevant memory blocks to reduce computation. The paper also introduces Dialogue NextLat, a training-only objective that predicts role-specific semantic transitions conditioned on paired user–system states. Experiments on F-Actor and Moshi show improved multi-turn dialogue quality and context consistency over the corresponding backbones.

Critical strengths
  1. The architectural idea is well motivated. Separating persistent dialogue-level semantic processing from continuous acoustic generation addresses a meaningful tension in full-duplex S2S systems. The shared causal timeline provides a concrete mechanism for making semantic history available during generation.
  2. The semantic improvements are supported across two backbones and several evaluation settings. The capacity controls, two-instance dialogue evaluation, and additional LLM judges strengthen the evidence that the approach offers useful semantic benefits. Selective bridging also retains most of the dense model’s quality at lower inference cost.
Critical weaknesses
  1. The advantage over alternative semantic-processing approaches is insufficiently established. Although Figure 1 outlines several design paradigms, the experiments do not adequately demonstrate why persistent selective bridging is preferable, particularly compared with Thinker–Talker architectures. The gains over simpler memory-access strategies are also modest.
  2. The semantic improvements do not establish a clear overall advantage for full-duplex dialogue. Full-duplex interaction results are mixed, with some degradation in turn-taking and interruption stop latency, alongside increased inference cost.
Questions for authors
  1. What explains the substantial TOR reduction for the Moshi-based model and the increased interruption stop latency for the F-Actor-based model? Do component ablations identify whether these regressions arise from semantic conditioning, training, or inference overhead?
  2. Is there evidence that the semantic improvements persist when users interrupt, overlap with the system, or correct previously stated information? In particular, how does persistent semantic memory recover from incorrect ASR tokens that have already been committed?
7 comments
AuthorsOpenReview ↗

Author Response 1: Summary

We thank the reviewer for the constructive assessment. We provide additional comparisons and analyses addressing the following concerns:

  1. Alternative semantic processing: A Thinker–Talker-style conditioning baseline and simulated-user evaluations further support the benefits of persistent semantic memory and selective retrieval.

  2. Full-duplex interaction: Updated interaction metrics and component ablations clarify the effects of semantic conditioning and reveal behavior-specific trade-offs in turn-taking and interruption handling.

  3. Semantic robustness: Event-conditioned analyses show that semantic improvements persist in dialogues containing interruptions, overlaps, and corrections. An additional ASR input-source comparison shows reduced sensitivity to causal ASR observations.

Detailed results and discussions follow below.

AuthorsOpenReview ↗

Author Response 2: Detailed Response

Comment 1 (W1): Comparison with Thinker–Talker and Alternative Semantic-Processing Approaches

We thank the reviewer for suggesting more direct comparisons with alternative semantic-processing architectures and simpler memory-access strategies. To better examine the role of semantic-to-generative communication, we implement a Thinker–Talker-style conditioning baseline under the F-Actor backbone, inspired by Qwen2.5-Omni [R1] and JoyAI-Talker [R2]. Both dual-stream configurations use matched backbone initialization, causal alignment, and training objectives. The Thinker–Talker-style baseline conditions generation on the latest final-layer semantic hidden state, whereas Bridge-Duplex uses layer-wise attention over persistent semantic K/V history.

For memory access, Selective Bridge Attention retains the most recent semantic block while enabling query-dependent retrieval of earlier blocks within the same four-block budget as the recent sliding-window baseline. This design aims to preserve access to relevant semantic information beyond the recent window, particularly as dialogue history grows. To further assess whether selective semantic memory access offers advantages over recent-window retrieval in longer dialogues, we introduce two complementary LLM-TTS-based user simulations: 40 open-ended conversations across 10 everyday topics inspired by Pipecat [R3] and FD-LHB [R4], and 200 staged task-oriented conversations across four categories adapted from FDB-v2 [R5]. Both settings use the same model-side inference pipeline and permit conversations of up to 20 minutes.

Table R1. Comparison of semantic-conditioning and memory-access strategies under the F-Actor backbone.

Model configurationDQ / CC ↑ (MTR-DB)DQ / CC ↑ (Behavior-SD)DQ / CC ↑ (Two-Instance)DQ / CC ↑ (Task Simulated-User)DQ / CC ↑ (Open Simulated-User)
F-Actor2.73 / 3.903.51 / 4.133.42 / 3.472.98 / 2.873.41 / 3.38
F-Actor w/ Thinker–Talker-style conditioning2.97 / 4.073.59 / 4.243.47 / 3.503.14 / 3.063.47 / 3.45
Bridge-Duplex w/ Recent Sliding Window3.02 / 4.013.66 / 4.283.46 / 3.643.28 / 3.393.43 / 3.51
Selective Bridge-Duplex3.03 / 4.093.68 / 4.313.49 / 3.673.32 / 3.473.47 / 3.65

As shown in Table R1, Selective Bridge-Duplex achieves higher CC and equal or higher DQ than the Thinker–Talker-style control across all five evaluation settings. This suggests that the benefits extend beyond stream separation to how persistent semantic information is communicated to generation. The simulated-user evaluations further show a clearer advantage of Selective Bridge Attention over recent sliding-window retrieval, particularly in context consistency. Despite its additional inference cost, these results provide further support for the benefits of query-dependent semantic memory access in interactive dialogue.

[R1] Xu, Jin, et al. "Qwen2.5-Omni Technical Report." arXiv preprint arXiv:2503.20215 (2025).

[R2] Bai, Yinhao, et al. "JoyAI-Talker: Full-duplex speech interactive large model built for empathetic voice agents." arXiv preprint arXiv:2608.01119 (2026).

[R3] Pipecat. Simulated Scenario Evaluation. https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/cli/agent_templates/AGENTS.md?utm_source=chatgpt.com

[R4] Kim, Yeonju, et al. "Why Full-Duplex Speech Models Collapse in Long Conversations." arXiv preprint arXiv:2610.32349 (2026).

[R5] Lin, Guan-Ting, et al. "Full-duplex-bench-v2: A multi-turn evaluation framework for duplex dialogue systems with an automated examiner." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2026.

AuthorsOpenReview ↗

Author Response 3: Detailed Response (Continued)

Comment 2 (W2, Q1): Full-Duplex Interaction Performance and Conditioning Analysis

We thank the reviewer for highlighting the full-duplex interaction regressions and their potential causes. To align with the official Full-Duplex-Bench protocol, we updated the v1.5 behavior evaluation using GPT-4o as the LLM judge and included the complete v1.0 and v1.5 results in Appendix C.5 of the revised manuscript. Here, we focus on the turn-taking and interruption behaviors raised by the reviewer, together with additional component ablations.

Table R2. Candor turn-taking and pause-handling performance on the Moshi backbone (Full-Duplex-Bench v1.0).

ModelTurn-Taking TOR ↑Pause-Handling TOR ↓
Moshi1.0001.000
Selective Bridge-Duplex (Moshi)0.6220.282

As shown in Table R2, Moshi frequently takes the floor even during user pauses, consistent with prior observations [R1, R2] that it often initiates responses before users finish speaking. In contrast, Bridge-Duplex substantially reduces premature takeovers during pauses, suggesting more conservative turn initiation, although it also misses some opportunities to respond at turn boundaries.

Table R3. User-interruption performance on the F-Actor backbone (Full-Duplex-Bench v1.5), including post-interruption behavior distributions (Respond, Resume, Uncertain, and Unknown) and timing metrics (Stop and Response Latency). Respond* is the desired behavior for user interruption in Full-Duplex-Bench v1.5.

ModelRespond* ↑Resume ↓Uncertain ↓Unknown ↓Stop (s) ↓Response (s) ↓
F-Actor0.0300.0100.0350.9251.0102.121
F-Actor w/ ASR0.1150.0350.0200.8301.7031.763
Dense Bridge-Duplex trained w/o Dialogue NextLat0.3800.1600.0300.4301.1451.352
Selective Bridge-Duplex w/o semantic conditioning at inference0.0000.0000.0100.9901.1941.655
Selective Bridge-Duplex (F-Actor)0.3250.1450.0150.5151.3951.825

As shown in Table R3, Selective Bridge-Duplex achieves higher Respond than both F-Actor and F-Actor w/ ASR, although its stop latency remains higher than F-Actor's. Removing semantic conditioning at inference substantially reduces Respond, suggesting its importance for addressing user interruptions. Meanwhile, the dense variant trained without Dialogue NextLat achieves higher Respond and shorter stop and response latencies than the proposed model. These results indicate that the evaluated configurations exhibit different interruption-handling trade-offs rather than uniform improvements.

Importantly, Full-Duplex-Bench v1.5 measures Stop Latency from the speech activity in the recorded user and generated audio, without explicitly accounting for wall-clock inference time [R3]. Therefore, the observed increase reflects the model's interruption timing under the benchmark and cannot be directly attributed to the additional computational overhead of the dual-stream architecture. This overhead may nevertheless affect responsiveness in real-time deployment.

Together, these results show behavior-specific improvements and regressions in full-duplex interaction. Bridge-Duplex primarily aims to improve multi-turn semantic modeling, while turn-taking and interruption timing remain important areas for further improvement. We have clarified these findings and remaining limitations in the revised manuscript.

[R1] Chen, Chen, et al. "Reinforcement learning enhanced full-duplex spoken dialogue language models for conversational interactions." Second Conference on Language Modeling. 2025.

[R2] Hu, K., et al. (2025) Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model. Proc. Interspeech 2025, 2715-2719, doi: 10.21437/Interspeech.2025-874

[R3] Full-Duplex-Bench, official timing evaluation implementation: https://github.com/DanielLin94144/Full-Duplex-Bench/tree/main

Reviewer_5kApOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2(+1)
Read review
Paper summary

Bridge-Duplex augments full-duplex speech models with a persistent text-based semantic stream alongside continuous speech generation. Bridge attention connects the streams, a learned selector retrieves relevant semantic-memory blocks, and Dialogue NextLat provides auxiliary supervision for semantic-state transitions. Experiments on F-Actor and Moshi report improved multi-turn dialogue quality and context consistency. Selective retrieval largely preserves dense-bridge performance while reducing its inference cost.

Critical strengths
  1. The semantic improvements are demonstrated with both F-Actor and Moshi, with additional evaluation through generated two-instance dialogues and multiple LLM judges. This provides useful evidence beyond a single backbone or evaluation setting.

  2. Selective bridging retains nearly all of the dense variant's semantic performance while reducing end-to-end RTF by approximately 18%. The retrieval diagnostics and implementation analysis help explain this result.

Critical weaknesses
  1. The motivation is more developed than the evidence that the added complexity is warranted. The system introduces a streaming ASR frontend, a separate semantic pathway, bridge attention, a learned selector, and an auxiliary state-prediction objective. While these choices are individually plausible, the experiments establish the effectiveness of the combination more clearly than the incremental value of its components. In particular, a matched single-stream baseline receiving the same causal ASR text would help distinguish better text evidence from the proposed semantic organization. Table 4 compares two NextLat variants but lacks an explicitly matched dual-stream baseline without NextLat. It therefore remains unclear how much of the benefit requires the proposed machinery rather than simpler conditioning or training changes.

  2. The benefit over simpler alternatives does not yet justify the practical trade-offs. The paper includes a useful 3B capacity control, so parameter scaling is not ignored. However, the selective model's gains over it are modest and not uniform, while RTF is higher (0.8287 versus 0.7480). In Table 3, learned selection also offers only small improvements over a recent window while running more slowly. Moreover, Table 1 shows a Moshi turn-taking TOR decrease from 1.000 to 0.622 and an F-Actor interruption stop-latency increase from 1.010 to 1.395 seconds. These results leave the overall advantage of the more elaborate architecture insufficiently established for its intended full-duplex application.

Questions for authors
  1. How does Bridge-Duplex compare with a matched single-stream model receiving the same causal ASR text and a comparable adaptation budget? What benefit remains from NextLat over the same dual-stream model without that auxiliary objective?

  2. Given the modest gains over the 3B and recent-window controls, what evidence establishes a favorable quality–cost trade-off? How should the turn-taking and interruption-latency regressions be weighed against the semantic gains?

7 comments
AuthorsOpenReview ↗

Author Response 1: Summary

We thank the reviewer for the careful assessment. To address the concerns about architectural complexity and practical trade-offs, we provide additional evidence on three aspects:

  1. Component contributions: Matched ASR-conditioned and dual-stream controls show that semantic organization improves multi-turn performance beyond ASR conditioning alone, while Dialogue NextLat provides consistent additional gains.

  2. Quality–cost trade-offs: To complement the ASR-sensitive Two-Instance evaluation, we conduct additional simulated-user evaluations of longer interactive dialogues. Despite a modest increase in inference cost, Selective Bridge-Duplex demonstrates clearer semantic advantages over simpler baselines in these longer-dialogue settings.

  3. Full-duplex interaction: We provide a more comprehensive analysis of Full-Duplex-Bench results, showing mixed, behavior-specific changes across full-duplex interaction metrics rather than uniform degradation, with improvements in interruption responsiveness alongside remaining turn-taking and stop-latency trade-offs.

We provide the experimental details and quantitative results below.

AuthorsOpenReview ↗

Author Response 2: Detailed Response

Comment 1 (W1, Q1): Disentangling the Contributions of Semantic Organization and Dialogue NextLat

We thank the reviewer for suggesting additional comparisons to clarify the contributions of semantic organization and Dialogue NextLat. We introduce an ASR-conditioned single-stream baseline and a Bridge-Duplex variant without Dialogue NextLat. All single-stream and dual-stream controls use the F-Actor backbone, identical adaptation data and training/validation splits, and the same number of scheduled training updates. For each configuration, we report the checkpoint with the lowest validation loss. F-Actor w/ ASR incorporates causal user-ASR text via additive embeddings with an auxiliary user-text prediction loss. All ASR-conditioned variants share the same text-delay and input-dropout augmentation during training.

Table R1. Component ablation of ASR conditioning, semantic organization, and Dialogue NextLat under the F-Actor backbone.

Model configurationTrainable params.User ASRSemantic streamDialogue NextLatDQ / CC ↑ (MTR-DB)DQ / CC ↑ (Behavior-SD)DQ / CC ↑ (Two-Instance)
F-Actor1.63BNoNoNo2.73 / 3.903.51 / 4.133.42 / 3.47
F-Actor w/ ASR1.89BYesNoNo2.80 / 4.033.57 / 4.212.90 / 2.93
Dense Bridge-Duplex trained w/o Dialogue NextLat2.87BYesYesNo2.98 / 4.053.66 / 4.313.43 / 3.57
Dense Bridge-Duplex3.05BYesYesYes3.04 / 4.093.72 / 4.353.51 / 3.68
Selective Bridge-Duplex3.06BYesYesYes3.03 / 4.093.68 / 4.313.49 / 3.67

As shown in Table R1, ASR conditioning alone improves the single-stream baseline on MTR-DB and Behavior-SD, but reduces performance in Two-Instance dialogue. The dual-stream model provides additional improvements over the ASR-conditioned single-stream baseline even without Dialogue NextLat. The Bridge-Duplex model with Dialogue NextLat achieves higher DQ and CC than its counterpart without NextLat across all three evaluation settings.

Two-Instance evaluates dialogues generated by two interacting instances of the same model rather than target responses conditioned on reference dialogue histories. Unlike the ASR-free baseline, ASR-conditioned models face an additional risk of recognition errors accumulating across both dialogue instances. To assess sensitivity to causal ASR input, Table R2 compares two user-text input conditions: directly passing the other instance’s generated agent-text and transcribing its generated speech using causal streaming ASR.

Table R2. Sensitivity to user-text input sources in Two-Instance dialogue evaluation.

Model configurationDQ / CC ↑ (Generator-text)DQ / CC ↑ (Causal ASR)ΔDQ / ΔCC (ASR − Generator-text)
F-Actor w/ ASR3.48 / 3.742.90 / 2.93−0.58 / −0.81
Selective Bridge-Duplex3.56 / 3.773.49 / 3.67−0.07 / −0.10

Table R2 shows that the single-stream baseline is considerably more sensitive to replacing generator text with causal ASR, whereas Bridge-Duplex maintains more consistent semantic performance across the two input conditions. Together, these results suggest that the benefits of Bridge-Duplex extend beyond additional ASR text evidence to the organization of semantic information and its integration into ongoing speech generation.

AuthorsOpenReview ↗

Author Response 3: Detailed Response (Continued)

Comment 2 (W2, Q2): Quality–Cost Trade-offs against Simpler Alternatives

We thank the reviewer for highlighting the quality–cost trade-offs against simpler alternatives. Table R3 shows that Bridge-Duplex improves semantic performance over the 3B baseline on MTR-DB and Behavior-SD, with comparable performance in Two-Instance dialogue. Unlike the ASR-free 3B baseline, ASR-conditioned models face an additional risk of recognition errors accumulating across both dialogue instances.

Table R3. Changes in semantic performance and E2E RTF relative to F-Actor (1B), including a direct comparison between Selective Bridge-Duplex and F-Actor (3B).

ModelΔ E2E RTF ↓ΔDQ / ΔCC ↑ (MTR-DB)ΔDQ / ΔCC ↑ (Behavior-SD)ΔDQ / ΔCC ↑ (Two-Instance)
F-Actor (3B)+0.216 (+41%)+0.19 / +0.07+0.12 / +0.12+0.08 / +0.16
Selective Bridge-Duplex+0.297 (+56%)+0.30 / +0.19+0.17 / +0.18+0.07 / +0.20
Δ (Selective − 3B)+0.081+0.11 / +0.12+0.05 / +0.06−0.01 / +0.04

Selective Bridge Attention builds on the principle of recency-based memory access while enabling query-dependent retrieval of earlier semantic blocks within the same retrieval budget. This design aims to preserve access to relevant semantic information beyond the recent window, particularly as dialogue history grows. To complement the ASR-sensitive Two-Instance evaluation and assess selective retrieval in longer dialogues, we introduce two complementary LLM-TTS-based user simulations: 40 open-ended conversations across 10 everyday topics inspired by Pipecat [R1] and FD-LHB [R2], and 200 staged task-oriented conversations across four categories adapted from FDB-v2 [R3]. Both settings use the same model-side inference pipeline and permit conversations of up to 20 minutes.

Table R4. Dialogue quality and context consistency under adapted Simulated-User evaluation.

ModelDQ ↑ (Task Simulated-User)CC ↑ (Task Simulated-User)DQ ↑ (Open Simulated-User)CC ↑ (Open Simulated-User)
F-Actor (3B)2.972.773.353.32
Bridge-Duplex w/ Recent Sliding Window3.283.393.433.51
Selective Bridge-Duplex3.323.473.473.65

As shown in Table R4, Selective Bridge Attention consistently improves DQ and CC over both baselines across the two simulation settings, with more noticeable gains in context consistency. Despite its additional inference cost, these results provide further support for the benefits of query-dependent semantic memory access in interactive dialogue.

[R1] Pipecat. Simulated Scenario Evaluation. https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/cli/agent_templates/AGENTS.md?utm_source=chatgpt.com

[R2] Kim, Yeonju, et al. "Why Full-Duplex Speech Models Collapse in Long Conversations." arXiv preprint arXiv:2610.32349 (2026).

[R3] Lin, Guan-Ting, et al. "Full-duplex-bench-v2: A multi-turn evaluation framework for duplex dialogue systems with an automated examiner." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Review Summary and Revisions

Dear Area Chair and Reviewers,

We sincerely thank you for your careful evaluation and constructive feedback. Your comments have helped strengthen the experimental evidence and clarify our contributions.

Strengths Highlighted by the Reviewers

  • Well-motivated architecture. Persistent semantic processing alongside continuous acoustic generation addresses a meaningful full-duplex challenge (xrUe), supported by stream-wise K/V analysis (kUqX).
  • Semantic gains and supporting evidence. Improvements hold across F-Actor and Moshi (5kAp, xrUe, kUqX), with additional evidence from Two-Instance dialogue and multiple LLM judges (5kAp, xrUe).
  • Efficient selective bridging. Selective bridging retains most of the dense model’s semantic performance while reducing inference cost (5kAp, xrUe).

Summary of Responses and Revisions

  1. Component contributions and alternative conditioning (5kAp, xrUe). We added causal-ASR single-stream, no-NextLat dual-stream, and Thinker–Talker-style controls under F-Actor. Results support benefits beyond ASR conditioning alone and additional gains from Dialogue NextLat.
  2. Quality–cost trade-offs (5kAp, xrUe). Simulated-user evaluations show DQ/CC gains over the 3B and recent-window controls. New ContextDialog and time-controlled recall tests support benefits of accessing earlier information, with remaining inference-cost trade-offs.
  3. Full-duplex interaction (5kAp, xrUe). Expanded Full-Duplex-Bench results and conditioning analyses clarify behavior-specific improvements and regressions.
  4. Semantic robustness (xrUe). Event-conditioned and ASR input-source analyses support semantic gains in dialogues containing interruptions, overlaps, and corrections, and reduced sensitivity to causal ASR.
  5. Broader speech capabilities (kUqX). We added Big Bench Audio, selected VoiceBench tasks, and an adapted MMAU-Speech setting. Bridge-Duplex improves over the corresponding conversation-adapted backbones.

Key textual revisions are highlighted in blue, and the listed locations refer to the manuscript or the corresponding author responses.


Reviewer 5kAp

ConcernResponseLocation
ASR conditioning and Dialogue NextLatWe added matched ASR-conditioned and dual-stream controls, showing that semantic organization improves multi-turn performance beyond ASR conditioning alone and that Dialogue NextLat provides consistent additional gains.Author Response 2; Secs. 5.1–5.2, Tables 2, 4.
Quality–cost trade-offsWe compared retrieval within the same four-block budget (CC: +0.08 task, +0.14 open) and added ContextDialog and time-controlled recall tests, showing larger recall gains at longer intervals.Author Response 3, Follow-up 5kAp; Tables 2–3, App. C.6, Table 11.
Interaction regressionsWe further analyzed the interaction results highlighted by the reviewer, showing mixed, behavior-specific changes across full-duplex metrics rather than uniform performance degradation.Author Response 4; App. C.5, Tables 9–10.

Reviewer xrUe

ConcernResponseLocation
Alternative semantic processingWe added a Thinker–Talker-style conditioning control and simulated-user evaluations, providing further evidence for the benefits of persistent semantic memory and selective retrieval.Author Response 2; Sec. 5.1, Table 2, App. C.6, Table 11.
Component effects and latencyWe further analyzed the interaction results and added component ablations, showing mixed, behavior-specific changes across full-duplex metrics rather than uniform performance degradation. We also clarified the distinction between benchmark stop latency and wall-clock inference overhead.Author Response 3; App. C.5, Tables 9–10.
Interaction events and ASR errorsWe added event-conditioned analyses, showing that semantic gains persist in dialogues containing interruptions, overlaps, and corrections. We also compared generator-text and causal-ASR inputs, showing reduced sensitivity to causal ASR observations.Author Response 4; App. C.7, Table 12.

Reviewer kUqX

ConcernResponseLocation
General capabilities beyond dialogueWe extended the evaluation beyond dialogue, showing that both Bridge-Duplex variants outperform their corresponding conversation-adapted baselines on all reported tasks.Author Response 2; App. C.8, Tables 13–14.

We hope this summary of the additional evidence and revision locations assists your final assessment. Thank you for your time and consideration.

sincerely,

The Authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.