acceptodds
Under review as a conference paper at ICLR 2027

Full-duplex Spoken Language Modeling with Discrete Diffusion

Abstract

We introduce Diffuplex, a full-duplex spoken language model based on discrete diffusion that achieves an unprecedented balance between linguistic intelligence and response speed. With thinking enabled, Diffuplex finalizes an average of 10.15 tokens per forward pass, achieving substantially higher output speed than comparably intelligent turn-based autoregressive models. Without thinking, it matches the first-audio latency of existing full-duplex models while delivering markedly stronger spoken intelligence. This capability is enabled by separating when to say from what to say, rather than compressing both into a single autoregressive timeline. During inference, Diffuplex continuously tracks the conversation through an autoregressive interaction policy, while independently generating and refining response content through parallel diffusion. Consequently, semantic computation is no longer bounded by the perception audio frame rate, allowing intelligence and responsiveness to improve simultaneously. Taken together, these results establish a new paradigm for spoken agents that can sustain real-time dialogue while devoting parallel computation to intelligent response generation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion language model, full-duplex model

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_qK1POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper first mentions current full-duplex models fail to exhibit strong linguistic and cognitive behavior compared to turn-based models. The authors argue that the major reason is the single one-dimensional autoregressive timeline, which causes gradient conflict between timing and semantics and frame-rate binding between 80 ms audio frame rate and Chain-of-Thought. They proposed a new architecture to decouple the "when to speak" and "what to speak" in a full-duplex model. Built on discrete diffusion language models, they use autoregressive mode to process incoming user audio and the prediction of model's turn taking. When the model decides to speak, it runs in diffusion mode for refining the response in an orthogonal timeline compared to the real timeline. They did an extensive evaluation of their model on common benchmarks and also several ablations to show that their model achieves a fast speed while being smart.

Critical strengths
  1. It is the first paper to integrate text diffusion models in the application of Full-duplex models to my knowledge.
  2. This paper proposed a good research question and direction to decouple "when to speak" and "what to speak" and this is tightly related to the current open problems in full-duplex models.
  3. They show that with diffusion for response generation, the decoded tokens per second is largely improved over other baselines.
Critical weaknesses
  1. The presentation of the overall model architecture is hard to follow and perhaps some information is missing.

    • Figure 2 is not easily understandable by readers. It is unclear from the figure how the model works during inference especially how it continually processes incoming 80ms chunks of user audio and formulates the response using the diffusion mode. The connection between the two modes is not easy to understand by looking at this figure.

    • The text description in the paper is more clear to me of how the AR mode and diffusion mode work together. If I'm understanding it correctly, there is single sets of weights for the backbone model (excluding the speech encoder and the TTS module). During inference, the AR mode is always on, it continually receives incoming user audio (80 ms) chunks and predicts the states <listen>, <speak>, <bos>, and <eos>. Once <bos> is sampled, it launches the "diffusion model", which run diffusion steps to generate the system response. So the entire system is like having 2 requests to the same model while one request is always on, the other only runs if the system response generation is triggered. However, I cannot find how these two requests are executed/scheduled during inference and I guess this also matters for latency.

  2. The problem formulation in the introduction is confusing and the discussion scope seems narrowed.

    • Confusing comparison: Starting form Line 61, the authors point out the pros and cons between "spoken language models achieve high intelligence but rely on external VAD" and "full-duplex models that unify when to say and what to say deliver fluid interactions but comes with severe linguistic and cognitive degradation". I think this is an oversimplified discussion. From my understanding the two reasons the authors brought up (Gradient conflict and Temporal Frame-Rate Binding) depend on whether the full-duplex system is cascaded or end-to-end. A turn-based model can be achieved by a cascading different model components or an end-to-end model. The same goes with full-duplex, we can achieve full-duplex behavior by cascading or using a single end-to-end model. I agree with the authors on the two reasons, but I think this depends more on whether the system is cascaded or not.

    • Lack of discussion for other recent Full-duplex system architecture: Continuing the system architecture design from the prior bullet point, there is another line of research that proposed to cascade two models to achieve a highly intelligent full-duplex system. They use one model for handling full-duplex interactions while the other model processes deep reasoning.

    • Not fully convinced of integrating diffusion in the full-duplex modeling.

      1. The author claims that the diffusion model for response generation decouples the conversation timing and response generation thereby avoiding the gradient conflicting issue. However, why particularly diffusion model? Aren’t any methods that separate into two models capable of working? In the appendix, Line 832-833, the author also tries using the AR mode for response generation and mentions that it sometimes even works better. If the advantage of using diffusion for response generation is about the speed, perhaps changing to another LLM backbone would help? I guess faster generation speed doesn't necessarily lead to better performance right? E.g. some models might be able to achieve a higher intelligence with a shorter reasoning trace, thereby still get an advantage with lower token decoding speed.

      2. Furthermore, the authors claim that the diffusion works orthogonal to the real physical time therefore solving the Temporal Frame-Rate Binding issue. However, I'm not clear how it entirely decoupled the two. From my understanding, while the diffusion model is formulating the response, it stays silent to the user. The real-time interaction still holds for the diffusion steps to finish.

  3. Thinking and Full-duplex seem to be exclusive in the experiments.

    It seems that there is no setting in the paper where the thinking and full-duplex are both on. I think this experiment is highly related to the paper's introduction. Line 69-71, full-duplex models can't allocate "elastic test-time compute (i.e., Chain-of-Thought).

  4. Full-duplex model baselines are outdated.

    The full-duplex baselines in the paper are classically full-duplex models (E.g. Moshi, Freeze-Omni...). However, they are somehow outdated. There are several recent full-duplex models that largely improve the intelligence while maintaining full-duplex responsiveness.

Questions for authors
  1. For Figure 1, why is the x-axis for no-thinking first audio latency, but for thinking on, the x-axis is output speed? For readers, I guess the first audio latency matters more than the decoding speed. I also cannot find the first audio latency for thinking on in the paper.
  2. With thinking on, Diffuplex seems to improve only a little on the intelligence score. What is the reason?
  3. Why there is no backchanneling results on FullDuplexBench?
4 comments
AuthorsOpenReview ↗

Response for Reviewer qK1P

Thank you for your insightful feedback. We clarify the following points and welcome further discussion.

  1. Distinguishing full-duplex models from full-duplex systems

Our discussion concerns full-duplex models that jointly learn conversational behavior and response generation, rather than full-duplex systems in general. Specifically, we target E2E models such as Moshi, where both functions are coupled within a frame-synchronous generative process. A cascaded system can also support full-duplex interaction but may delegate turn-taking to a separate controller.

Our concerns about conflicting training objectives and temporal frame-rate binding apply when conversational control and semantic generation are coupled within the same prediction process; they are not universal consequences of full-duplex interaction. Diffuplex addresses this coupling by learning the two functions separately within a shared backbone--like all baselines in this work.

  1. Coverage of recent full-duplex architectures

Our comparison already includes FLAIR, Nemotron-VoiceChat, and Lychee-FD—all 2026 works, shown in Figure 1.

We welcome suggestions of additional relevant open-source full-duplex models and would be happy to evaluate them during the discussion period, provided checkpoints and reproducible inference implementations are available.

  1. Why diffusion if AR generation sometimes performs better?

The primary motivation for diffusion is fast response generation, rather than an assumption that it must outperform AR decoding in response quality. In our updated experiment, Diffuplex generates an average of 1,873 text tokens/s on a single GB300 GPU, approximately 4.8× the measured throughput of a similarly sized AR model. Diffusion finalizes multiple response tokens per backbone forward pass: 13.0 tokens on average, compared with 1.0 for standard AR decoding.

For spoken interaction, response quality must be considered alongside generation efficiency and latency. Our objective is to preserve strong linguistic capabilities while substantially accelerating generation. Text throughput and first-audio latency are distinct metrics: the latter also includes turn-taking decisions and speech synthesis, which is why we evaluate them separately. The backbone’s competitive AR generation mode provides an alternative operating point.

  1. What does temporal frame-rate binding mean?

Our concern is specifically about tying semantic generation to the real-time audio clock. In Moshi, the text stream is aligned with audio frames at 12.5 Hz. Inserting 125 additional reasoning tokens into this frame-synchronous stream would require at least 125 steps, or approximately 10 seconds. A faster GPU does not remove this alignment constraint without changing the inference formulation. This is what we mean by restricted “elastic test-time compute.”

Diffuplex retains an 80-ms interaction-policy update interval, but its response generation is not paced by incoming audio frames. Once triggered, diffusion generates and refines reasoning and answer tokens independently of that clock.

We agree that cascaded systems with separate text generators can also avoid frame-rate binding. Diffuplex additionally offers substantially higher measured text-generation throughput than the evaluated AR counterparts, reducing the generation cost of reasoning, although enabling thinking still adds latency.

  1. Why is the intelligence gain from thinking relatively modest?

Our training recipe prioritizes low-latency spoken interaction, rather than maximizing VoiceBench scores. Many spoken requests do not require extensive reasoning, and long thinking traces can introduce unnecessary delays. We therefore train 70% of Stage-2 responses using direct answers without explicit thinking content, strengthening the model’s ability to respond without prolonged deliberation. We beleive such kind of concise model could bring better user experience for spoken interaction.

  1. Architecture presentation and inference workflow

Thank you for this suggestion. The temporal interaction between the two modes is difficult to convey fully in a static figure. We provide a workflow animation and a recorded inference demo in the supplementary material, with the user’s voice muted to preserve anonymity.

In the revised manuscript, we will expand the appendix with annotated snapshots from the animation to illustrate the inference workflow step by step.

Thanks for the authors' reply.

There is still something that is unclear to me.

After reading the authors response, my understanding the argument is two-fold for this paper (correct my if I'm wrong)

  1. Decoupling conversational behavior from response content generation can improve the model’s performance.
  2. Using diffusion for response generation can enhance the decoding speed.

I agree with the authors on the first point. However, for the current version of the paper, it seems misleading to the readers. The two points are not disentangled. What I meant is that diffusion is not the only way to decouple conversational behavior from response content generation. The author also agreed in the reply that cascaded full-duplex system is also an option.

We agree that cascaded systems with separate text generators can also avoid frame-rate binding.

I would suggest mentioning cascaded systems in the intro and related works so that the readers won't get misled.

For the second point, I also agree with the authors that decoding speed is an advantage of diffusion models. However, in the context of full-duplex conversation, most of the time, the response is short. Under typical conversation, how much speed advantage do we get from using diffusion compared to AR? And how does the quality of spoken response compare? And since we are operating in a real-time full-duplex conversation, the speaking is in streaming mode. Even if we need a long response, we don't have to wait till the AR model finishes the entire response generation. Its output can be streamed to the user while the response is still autoregressively generating.

I think it is important to include the AR ablation (generate the response token by token) which is currently mentioned briefly in the appendix with no evaluation numbers. This will be a strong result for using diffusion because this is the apple-to-apple comparison while any other Full-Duplex model baselines all differ in model sizes, training data... etc. Given that the author already mentioned such an experiment in the original manuscript, I think they do not need to run additional experiments on this.

The other benefit of using diffusion is for thinking, which is mentioned implicitly in the manuscript. The authors also mention this explicitly in the rebuttal.

Diffuplex additionally offers substantially higher measured text-generation throughput than the evaluated AR counterparts, reducing the generation cost of reasoning, although enabling thinking still adds latency.

However, "full-duplex" and "thinking" seems exclusive in the current proposed system. (Please correct me if I'm wrong. I've raised this question in the review but it seems like the author missed it.) This further undermines the advantage of using diffusion in the full-duplex conversation.

Other points:

  1. The author mentioned FLAIR is evaluated as a baseline, however, it only shows up in the first graph on the first page but no detailed results on the main table (Table1).

  2. Regarding other baseline full-duplex architectures, I think "MiniCPM-o 4.5" (https://arxiv.org/pdf/2604.27393) is one suitable choice (an end-to-end model)

    From my understanding, the current Diffuplex run two processes (two models inference with different decoding logic) when generating the response but there is only a same set of model weights. Given this fact, I think Diffuplex is in between the end-to-end and cascaded full-duplex models. So I think even cascaded full-duplex systems are worth putting in the table as references.

    Given that the deadline for rebuttal is approaching, I think it is reasonable that the author might not have time to add more suitable baseline/references. And comparing to the other points I raised in the response, adding other model baselines and references is not the top priority because it is hard to have a fair comparison directly. That said, it is still good to have more stronger Full-Duplex models in the main comparison table.

  3. The back-channeling results on FullDuplexBench is still missing

  4. The detail for the inference logic is still missing raised in W1

    However, I cannot find how these two requests are executed/scheduled during inference and I guess this also matters for latency.

  5. Is Full-Duplex and thinking exclusive for Diffuplex?

    The results in the paper seem exclusive.

  6. First-audio latency with thinking on is still missing.

    I understand text throughput and first-audio latency are distinct metrics as mentioned in the authors' response. But why aren’t both metrics reported for both thinking on and off?

AuthorsOpenReview ↗

Response for Reviewer qK1P (Part-1)

Thank you for the follow-up and for identifying the comparisons that would most directly clarify our contribution. We address your points below.

  1. Separating decoupling from diffusion

We agree that decoupling conversational control from response generation does not require diffusion. Our contribution combines two complementary choices: learning conversational control separately from response generation, and using diffusion to accelerate the latter within a shared backbone. We will revise our realted work and introduction according to your suggestions, and thanks again for your comment.

  1. Same-model AR comparison and the practical value of faster generation Thank you for emphasizing this comparison. The AR compairson results are: | AR decoding | Alp. | Com. | Wil. | SDQA | MMSU | OBQA | BBH | IFE. | Adv. | Overall | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | Non-thinking | 95.40 | 87.60 | 90.00 | 64.60 | 73.80 | 87.90 | 86.50 | 80.30 | 99.60 | 85.10 | | Thinking | 94.09 | 87.00 | 89.72 | 62.93 | 82.27 | 94.07 | 88.90 | 86.34 | 98.85 | 87.13 |

We agree that AR can hide much of the generation time for short spoken answers. However, spoken agents are also expected to support task-oriented interactions involving reasoning and tool use (like TUA-voice). Internal reasoning or a structured tool call may need to finish before the corresponding action or spoken answer can begin, so this computation cannot always be hidden behind audio playback. The same-model first-audio measurements below directly examine this setting. We have number for reference in the next quesiton.

Serving efficiency is another consideration: a company serving many users from a fixed GPU pool must consider the computation required per interaction, not only whether one conversation can stream in real time. More efficient generation could provide capacity for additional sessions. We present this as a deployment motivation, rather than claiming that single-request TPS establishes multi-user serving throughput.

  1. Compatibility of thinking and full-duplex interaction

Thinking and full-duplex operation can be enabled simultaneously in Diffuplex, as demonstrated in our supplementar video. The AR interaction policy continues monitoring the user while the response generator produces its thinking trace and answer.

We now report the measured first-audio latency with thinking enabled, using the Fisher interaction-policy configuration:

Response decodingThinkingFirst-audio latency
DiffusionOff0.396 s
DiffusionOn0.711 s
AROn1.689 s

Enabling thinking adds approximately 0.32 s to diffusion’s first-audio latency. With thinking enabled, AR decoding takes approximately 0.98 s longer than diffusion in the same model. Comparing its thinking-on latency directly against other AR models with different reasoning lengths would therefore confound decoding efficiency with reasoning behavior. The same-model comparison is more informative, although generated reasoning lengths can still differ between decoding modes.

We also wish to explain why our original full-duplex comparison used the non-thinking configuration. All full-duplex baselines in that comparison were evaluated without explicit thinking. Enabling thinking only for Diffuplex would create an asymmetric comparison. Nevertheless, we agree that reporting thinking-on quality and latency together provides a useful additional operating point, the new measurements make the trade-off explicit: thinking increases first-audio latency, but diffusion substantially reduces that increase relative to AR decoding while achieving similar response quality.

AuthorsOpenReview ↗

Response for Reviewer qK1P (Part-2)

  1. Inference scheduling

The AR policy and diffusion generator share one copy of the backbone weights on a single GPU and execute alternately, rather than concurrently, with separate KV caches. The scheduler prioritizes the policy’s single-step state prediction at each 80-ms update, with the listening-mode adapter enabled. Between policy updates, it advances diffusion generation with the adapter disabled. Thus, the two modes share GPU execution time; policy updates take priority, while diffusion uses the intervening compute budget.

The policy’s decisions control response generation. A <bos> decision triggers context prefill and diffusion decoding. The first committed prefix is immediately passed to TTS while the remaining text continues denoising; subsequent committed text is also streamed to TTS. First-audio latency therefore includes the policy’s decision delay, context prefill, prefix generation, and synthesis of the first audio packet. When the policy predicts <eos> in response to a user interruption, the system stops playback and cancels unfinished response generation. The next response is conditioned on the updated context, including the user’s interruption.

The current implementation retains all generated response text in the persistent generation context, including text that may not have been played before an interruption. This is a limitation of the current context-management implementation: text generation runs ahead of playback, and we do not yet align playback progress with individual text positions. We plan to combine TTS word-level timestamps with the playback position to retain only the spoken prefix and discard the unplayed remainder.

  1. Additional baselines and cascaded systems

Thank you for recommending MiniCPM-o 4.5. We have completed its VoiceBench evaluation and report the results below:

ModelAlp.Com.Wil.SDQAMMSUOBQABBHIFEvalAdv.Overall
MiniCPM-o 4.584.071.871.668.3566.6987.6955.0080.5998.8576.06

The three open-ended QA scores are normalized to a 0–100 scale. We will include these results in Table 1. The omission of FLAIR’s detailed results was an oversight, which we will also correct.

Regarding conventional ASR–AR LLM–TTS cascades, their linguistic capability largely comes from the underlying turn-based AR model, while transcription errors can introduce additional losses. Their first-audio latency additionally depends on speech recognition, turn detection, and synthesis. Consequently, we would generally expect them to approach the capability of their turn-based backbone, rather than exceed it, and not necessarily match the low latency of tightly integrated models such as Moshi.

Our existing comparisons therefore provide complementary reference points: strong turn-based AR models for linguistic capability, and models such as Moshi for low-latency interaction. This does not replace evaluating a complete cascade, nor do we claim these reference points constitute strict performance bounds.

We are happy to add the relevant cascaded systems you recommend, provided their implementations are available for reproducible evaluation. We will also explicitly discuss cascaded full-duplex systems in the introduction and related work.

  1. Backchanneling

The submitted version did not explicitly train or evaluate agent-generated backchanneling as we view turn-taking and barge-in are more important. To address your concern, we retrained Stage 3 (w. Fisher) with an additional <bc> control token. We annotated backchannel segments in the Stage-3 interanl conversational data, which had previously been treated as silence, and used them to supervise the new state.

Due to time constraints, the Stage-2 response generator remains unchanged.* When <bc> is predicted, we prompt it to generate a 1~4 words acknowledgment of the user's statement. The generated backchannel is synthesized for playback but is not appended to the persistent response-generation KV cache.

The backchanneling results are:

ModelTOR ↓Freq ↑JSD ↓
Moshi1.0000.0010.957
Freeze-Omni0.6360.0010.997
Gemini Live0.0910.0120.896
GPT Realtime 2.10.0190.00990.962
Qwen Audio 3.0 flash0.0180.0020.982
Awaken0.1450.0730.753
Diffuplex0.0550.0160.830

We also observed a trade-off: after adding backchanneling, turn-taking TOR-v1 to 93.3% from 100%. We have not had sufficient time within the rebuttal period to tune this balance further.

Reviewer_TyVzOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tackles the task of full duplex speech model i.e. a speech-to-speech model that can predict silences seemlessy, effectively able to model conversational aspects such as interruption, change of turns, backchanneling etc. without external VADs or other modules. The proposed method decouples the semantic generation, done by a diffusion LLM, from the conversational objective handled by a lightweight autoregressive policy which determines when to speak. The proposed model uses NVIDIA's Parakeet as audio encoder and DiffusionGemma as the LLM backbone. The first stage finetunes only the audio-text alignment layers. The second stage finetunes the model on on speech instruction samples using a frozen version of the base DiffusionGemma as a teacher anchor. The final stage finetunes the model for conversational behavior. The inference is similar to that of DiffusionGemma with a few optimizations to make it run in streaming.

Critical strengths
  • Interesting paradigm for voice agents that fall somewhere between full-duplex models like Moshi or PersonalPlex and cascaded models
Critical weaknesses
  • "Full Duplex" definition: The main weakness is the placement of the paper in full-duplex literature:

    • (a) Unlike Moshi or PersonaPlex, Diffuplex can only decide when to speaks, but it cannot speak at the same time as the user, preventing it to model conversational behaviors such as backchanneling or lauging etc. The output side is also based on a TTS so cannot easily model things like prosody and expressivity as it is only conditioned on text.
    • (b) Similarly, the introduction suggests that recent full duplex models "exhibit severe linguistic and cognitive degradation" because they "unify when to say and what to say". However among the full-duplex baselines used in the paper: Moshi was trained on top a 7B model from 2 years ago, Nemotron VoiceChat starts from an 11B Nemotron Nano 2. In comparison Diffuplex starts from a 27B (4B active) DiffusionGemma model. To make such conclusions it would be more informative to report how much each model's performance drops compared to their LLM backbone baseline (closer to Table 3) rather than report absolute "intelligence" numers
    • (c) To be fully complete, it would also be interesting to compare Diffuplex to the full cascade alternative ( ASR, DiffusionGemma backbone, CosyVoice TTS)
    • (d) Moshi and PersonaPlex are not compared to in terms of latency e.g. in Figure 3
  • On tying the model to a physical framerate: While I agree it is a design limitation to auto-regressive backbones, I find that the current paper does not illustrate enough how we could benefit from not having this limitation

  • (Minor) When it comes to decoupling "intelligence" from "conversationality", I think two citations are missing:

    • "KAME: Tandem Architecture for Enhancing Knowledge in Real-Time Speech-to-Speech Conversational AI"
    • "MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Models"
  • (Minor) I found some part of the writing to be a bit confusing

    • "Response canvas" to me evoke image generation much more than speech
    • Figure 2 mentions a <bos> token which is not on the figure
Questions for authors
  • In Table 1, how much of the intelligence improvement is due to the new DiffusionGemma backbone ? It would also be interesting to introduce some kind of latency numbers to the table to highlight the real-time (or not) property of each model
  • In Figure 3, why are there no latency comparison to Moshi and PersonaPlex ?
  • Why is the experiments from line 374-377 not in the main Table ? It sounds like there could easly be a Nemotron-Labs-Diffusion-8B row in the table and all other tables
3 comments
AuthorsOpenReview ↗

Response for Reviewer TyVz

Thank you for these thoughtful and constructive comments! We have organized your concerns into six points and address them below.

  1. Full-duplex capability and expressive speech generation

We would like to distinguish two aspects: speaking while listening, including backchanneling, and generating expressive speech.

Diffuplex’s architecture does not require the user to remain silent during response generation or playback. The AR policy continues monitoring incoming speech while diffusion generates the response and TTS produces speech. Whether to retain or yield the conversational floor is a learned policy decision.

Agent-generated backchanneling could be supported by extending the Stage-3 control vocabulary with a <backchannel> state and training the policy to predict appropriate insertion points. This event could trigger a short, appropriately prompted response without requiring further fine-tuning of the Stage-2 diffusion generator. We distinguish this proposed extension from the behaviors explicitly trained and evaluated in the current version.

Regarding TTS, your comment identifies a trade-off we considered. Directly modeling multiple audio codebooks with diffusion would provide a more unified formulation. We instead chose a modular TTS interface to benefit from rapidly improving speech synthesis models, including their latency, synthesis quality, and controllability. A TTS-based output interface does not inherently preclude paralinguistic expression. For example, Chatterbox-Turbo supports tags such as [cough], [laugh], and [chuckle], which a language model can generate as synthesis controls. This illustrates an extension enabled by the modular design, rather than an expressive capability established by our current experiments. We will clarify this distinction and discuss emotion-conditioned speech generation as future work.

  1. Backbone capability versus capability preservation

The text backbone largely determines the intelligence ceiling; the modeling approach determines how much capability is retained when adapting it to speech. Our VoiceBench results illustrate this:

  • DiffusionGemma: 89.2 with text input → 86.9 with speech input after Stage 2.
  • Nemotron-Labs-Diffusion-8B: 79.2 with text input → 78.7 with speech input, almost no loss.

We did not include Nemotron-Labs-Diffusion-8B as a primary comparison because its TPF decreased after Stage 2, removing its decoding-speed advantage over AR models.

Similar capability preservation is achievable when adapting AR text models into turn-based speech models, as in Qwen-Omni and Nemotron-Omni.

The larger capability loss appears when these language models are adapted for full-duplex interaction. The full-duplex baselines in our comparison score below 60, despite their text backbones scoring above 80. We attribute this gap in part to coupling user audio, generated text, and generated audio within the same AR process. Diffuplex separates conversational control from response generation to reduce this interference. This is also one reason we keep speech synthesis outside the diffusion backbone.

Back to this work, DiffusionGemma is indeed an important part of our results. But our primary contribution is to brings the capabilities of duffioson langude models to spoken interaction while minimizing capability loss, which is not specific to DiffusionGemma, and we expect it to apply to future diffusion backbones as well.

  1. Comparison with an ASR–DiffusionGemma–TTS cascade

In general, a transcript-only interface discards information not represented in the transcript, including potentially useful emotional and paralinguistic cues. ASR also introduces an additional recognition and stabilization stage before downstream response generation. Streaming ASR can overlap computation, but its endpointing and transcript-commitment behavior can still affect first-audio latency. Meanwhile, the results of Diffuplex (text input) in Table 3 can be viewed as upperbouand of cascade, asumming all WER is 0.

  1. Why are Moshi and PersonaPlex not included in Figure 3?

We omit Moshi and PersonaPlex because Figure 3 compares decoding efficiency to show the adavanatge of diffusion compared with AR, whereas Moshi-style's text generation is tied to real-time audio frames. At 12.5 Hz, Moshi and PersonaPlex dilevry fixed 1 TPF and 12.5 TPS (text-token per second, including padding). Faster GPUs cannot remove this constraint without changing their frame-synchronous inference formulation, which is precisely the temporal frame-rate binding discussed in our paper.

(Please see the next comment)

AuthorsOpenReview ↗

Response for Reviewer TyVz (2)

  1. Practical benefits of removing frame-rate binding

We beleive the question 4 potentially provides the reson for this question. Our additional hardware results illustrate the compute-scaling opportunity: Diffuplex averages 1,872 text tokens/s on GB300 and 1,045 text tokens/s on H100, substantially exceeding the measured throughput of similarly sized AR counterparts. These are text-decoding rates, rather than end-to-end first-audio latency measurements.

This separation is especially relevant to future spoken agents that might serve as a interaction front-end who requires tool calling or delegate work to a larger text agent. Their internal thinking process should be fast and does not need to dilvery to user, thus removing frame-rate binding is necessary.

  1. Related work and presentation

Thank you for recommending KAME and MoshiRAG. We will incorporate both and clarify their relationship to our approach.

We will also define “response canvas” explicitly as a block of text-token positions jointly refined during diffusion, and make the <bos> trigger more visible in Figure 2. Our supplementary material includes a workflow animation and a recorded inference demonstration illustrating how the two modes interact.

Thank you for your response and clarification After reading the rebuttal and other reviews, I am inclined to keep my score: While I lean positive on the paper, I do think it would benefit from a clearer placement and comparison in the landscape of full-duplex systems/models

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.