acceptodds
Under review as a conference paper at ICLR 2027

DiffuPlex: Accelerating Full-Duplex Spoken Dialog Models via Rolling Masked Diffusion

Abstract

Recent full-duplex spoken dialog models enable simultaneous listening and speaking, but fine-grained models still advance their backbone autoregressively at every interaction frame. We introduce DiffuPlex, a rolling masked diffusion framework that reduces this sequential computation by predicting multiple future user and assistant frames in a single backbone wake. DiffuPlex consumes only a confident prefix of each predicted future while interaction continues at the original frame rate. As user speech arrives, it checks the corresponding user predictions and, when the interaction diverges, preserves already played assistant content while revising only the unplayed future. We consider two inference policies over the same predictor: DiffuPlex-LISTEN consumes multiple future frames when they predict assistant silence, whereas DiffuPlex-SPEAK can also consume predicted assistant speech. Across full-duplex interaction and spoken-language evaluations, DiffuPlex substantially reduces sequential backbone computation while largely preserving interaction behavior and general capability. DiffuPlex-LISTEN and DiffuPlex-SPEAK achieve and deployment-path wall-clock speedups and and Core LM speedups, with all measured backbone invocations completing within the 80ms interaction interval. Human evaluation shows that LISTEN preserves speech naturalness and conversational quality, while SPEAK retains conversational quality with some degradation in speech naturalness.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spoken Dialog Model, Speech LLM, Diffusion LLM, Full-Duplex Spoken Dialog Model, Full-Duplex Speech LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fkjsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Authors propose DiffuPlex, the method for full-duplex dialogue model. Rather than the agent listening to the user and responding autoregressively, whenever the user fills in one frame, it predicts future user and assistant frames within a fixed block, and if the user input diverges from this prediction, it revises only the subsequent part.

Critical strengths
  • The idea of predicting future user and assistant frames within a block and revising only the subsequent portion when the observed user input diverges from the prediction is interesting.
  • The paper examines runtime alongside conversational behaviors such as turn-taking and backchanneling, rather than reporting runtime results alone. Although the interpretation and practical significance of these results need further clarification, considering both aspects is a positive feature.
Critical weaknesses
  • In lines 43–44, you say “update relatively coarse chunks,” but it is unclear what this is being compared with and roughly how many Hz “coarse” refers to. It would be better to mention even a simple number.

  • The ratio between successful predictions and cases requiring revision seems likely to vary depending on whether the user is having a normal conversation and on the evaluation dataset. However, the 60-clip runtime subset mentioned in the caption of Table 2 is not sufficiently characterized in terms of turn-taking, pause handling, interruption, and backchanneling.

  • The reduction in Time and increase in speed offline probably assume running as fast as possible rather than actually conversing with a person, and I question the necessity of this result. In highly constrained situations such as on-device settings, I understand doing this in preparation for cases where computation cannot finish within 80 ms. However, since the GPU used is an RTX A6000, the benefit obtained by reducing computation by that amount seems very marginal. It seems necessary to strengthen the argument for this need, for example, by showing that it helps when running on a CPU or NPU. In particular, looking at the Wake behavior column in Table 2, the p99 time is actually slower for DiffuPlex, and RL-Seamless also appears to have no cases exceeding 80 ms. If so, I wonder whether there is no advantage in terms of the timing aspect mentioned in the introduction.

  • To be honest, I had difficulty understanding Figure 2. First, it would have been easier to understand that the blue, green, yellow, and purple elements in (a) are codec streams if cb0, cb1, etc. were shown on the left, as in Figure 4. Also, it would be easier to understand if the “Interaction unfolds” panel showed what role the stored predictions U1 and U2 play during revision.

  • You say that Figure 3 shows efficiency–interaction, but it is unclear whether turn-taking latency and backchannel frequency are really metrics related to efficiency rather than metrics showing the model’s accuracy. For turn-taking latency, to put it plainly, couldn’t it be achieved by fine-tuning PersonaPlex on data with fast turn-taking, and couldn’t backchannel frequency be achieved by fine-tuning on data with frequent backchanneling? Also, in Figure 3(c), it is unclear what the y-axis means. Even if that were understood, the score differences among AR, Silero, the two Smart Turn models, and DiffuPlex are small, so I question whether they are significant.

  • I am not an expert in diffusion LLMs, but from the perspective of someone in the speech field, I am not sure whether DiffuPlex’s methodology really uses diffusion. Rather, it uses a BERT-style masking–prediction method, and considering that it is distilled from an autoregressive model, I wonder whether it would be more appropriate to categorize it as speculative decoding.

  • Since Full-Duplex-Bench v1.0 used GPT-4o as the Interruption Judge, results measured using GPT-4o also seem necessary to make the same comparison with other papers.

Questions for authors
  • My understanding is that, rather than the agent listening to the user and responding autoregressively, whenever the user fills in one frame, it predicts future user and assistant frames within a fixed block. If the user input diverges from this prediction, it revises only the subsequent part. Is draft correctness judged based on whether it diverges, rather than on an exact match/mismatch?
  • What is the ratio between cases where prediction succeeds and actually reduces the time and cases where revision is ultimately needed? Could you provide statistics on turn-taking, pause handling, interruption, and backchanneling, or a few actual data examples, for the 60-clip runtime subset mentioned in the caption of Table 2?
  • If revision is ultimately needed, does it take more time than autoregressive generation? It would be easier to understand if you showed the case where everything is done autoregressively (worst case) and the case where everything succeeds through diffusion (best case) on some of the evaluation datasets.
  • Is latency actually a problem for models like PersonaPlex? Is the actual response time slow enough to be inconvenient? Given the Wake behavior results in Table 2, what advantage does DiffuPlex provide in terms of the timing aspect mentioned in the introduction?
  • How are turn-taking latency and backchannel frequency related to efficiency rather than accuracy? What does the y-axis in Figure 3(c) mean, and are the small score differences among AR, Silero, the two Smart Turn models, and DiffuPlex significant?
  • If there is a paper from the diffusion LLM literature that you have substantially drawn on, I would appreciate it if you mentioned it in the methodology, for example, “we adopted ~ et al.”
  • Could you also provide the results in Table 1 measured using GPT-4o as the Interruption Judge, as in Full-Duplex-Bench v1.0?

Since I did not use AI while reviewing, the answers to my questions may be somewhere in the appendix. If so, please let me know by mentioning the section numbers, and I will consider changing my score.

12 comments

Thank you for the careful reading and detailed feedback. Your comments prompted additional experiments and helped us clarify the method and its practical benefits. We address related weaknesses and questions together below. Manuscript changes are highlighted in red.

W7 & Q7. Evaluation with GPT-4o as the interruption judge.

Thank you for this suggestion. We rescored the original Table 1 outputs and transcripts using gpt-4o-2024-08-06, retaining the official FDB v1.0 evaluation prompt and scoring procedure.

ModelGPT-4o interruption score ↑
Moshi3.529
PersonaPlex4.543
PersonaPlex-RL-Seamless4.578
DiffuPlex-LISTEN4.591
DiffuPlex-SPEAK4.648

Both DiffuPlex policies achieve GPT-4o scores similar to the AR backbone. These results support our finding that reducing backbone calls largely preserves interruption-response appropriateness.

We have updated all FDB v1.0 interruption-judge scores in the manuscript, including the main comparisons, ablations, and corresponding paired statistics. All corresponding scores in the remaining responses, including additional experiments, also use gpt-4o-2024-08-06.

W3 & Q4. What practical benefits does acceleration provide when the backbone already meets the 80ms deadline?

Your question motivated us to examine how computational savings affect real-time operation. We conducted additional experiments with input frames arriving every 80ms, measuring resource consumption and evaluating multiple conversations on the same GPU.

  • Single-session timing and resource consumption.

    As you point out, RL-Seamless already meets the 80ms deadline for a single session on the A6000. DiffuPlex predicts multiple future frames at each wake, making individual wakes more expensive but less frequent. This explains the higher wake-frame p99 in Table 2: 46.85ms for RL-Seamless versus 61.34ms for DiffuPlex-LISTEN and 61.41ms for DiffuPlex-SPEAK. All measured wakes remain within 80ms, while total computation over the conversation decreases.

    Under real-time pacing on the 60-clip runtime subset, GPU utilization decreases from 58.0% for RL-Seamless to 38.0% for DiffuPlex-LISTEN and 34.4% for DiffuPlex-SPEAK. GPU energy consumption decreases by 16.1% and 19.9%, with 95% confidence intervals of [12.6,19.4]% and [17.4,22.5]%, respectively. These savings provide a practical benefit even when a single session already meets its deadline.

  • Multiple conversations on the same GPU.

    In deployment, multiple conversations may share a GPU. We evaluated all 727 FDB v1.0 clips with 3 concurrent sessions on an RTX PRO 6000 Blackwell Max-Q (96GB) and 2 on an RTX A6000. Each session uses its own model instance, with one run per setting.

    If a frame’s output is not ready within its 80ms interval, the resulting audio is buffered for later playback. All systems use the same playback policy. We report the official Smooth Turn Taking metrics (TOR and latency) and User Interruption metrics (TOR, Judge, and latency), measured from the same playback outputs.

    GPU / sessionsModelTurn TOR ↑Turn latency ↓Int TOR ↑Int Judge ↑Int latency ↓
    PRO 6000 Max-Q / 3RL-Seamless0.8070.894s0.9854.5332.233s
    PRO 6000 Max-Q / 3DiffuPlex-LISTEN0.9830.191s0.9904.5511.524s
    PRO 6000 Max-Q / 3DiffuPlex-SPEAK0.9920.203s1.0004.6350.914s
    A6000 / 2RL-Seamless0.8570.832s0.9954.5182.064s
    A6000 / 2DiffuPlex-LISTEN1.0000.226s0.9904.5811.609s
    A6000 / 2DiffuPlex-SPEAK1.0000.193s0.9954.6180.759s

    Latency measures the interval from the end of the user’s utterance to the assistant’s first response word in the corresponding task. Judge evaluates interruption-response appropriateness using GPT-4o.

    On the PRO 6000 Max-Q, increasing from 1 to 3 sessions raises turn-taking latency from 0.101s to 0.894s for RL-Seamless, from 0.144s to 0.191s for DiffuPlex-LISTEN, and from 0.156s to 0.203s for DiffuPlex-SPEAK. The increase is therefore 793ms for RL-Seamless and 47ms for either DiffuPlex policy. Thus, the savings help maintain more stable response times under concurrent load, with interruption-appropriateness scores similar to RL-Seamless at the same load.

  • CPU/NPU deployment.

    We also attempted CPU inference on a dual-socket Intel Xeon Gold 6348 server with 56 physical cores. In the tested configuration, even a single AR decoding step exceeded 80ms, preventing frame-by-frame inference at the original interaction rate. Enabling efficient full-duplex interaction on CPUs and NPUs remains a valuable research direction.

Section 5.2 now reports the resource savings and concurrent-session protocol, with complete FDB v1.0 automatic metrics in Table 3. We appreciate your guidance in helping us substantiate these practical benefits beyond offline speedups.

W2, Q2 & Q3. Runtime-subset coverage, draft use, and revision costs.

Thank you for these questions. We apologize that the runtime subset was not sufficiently explained. We provide its composition, draft-use statistics, and a full-benchmark timing check below.

  • The 60-clip subset and draft use.

    For controlled timing, we ran systems sequentially on one A6000, with no other experimental jobs on the server. We used 60 clips to finish these isolated measurements sooner: 15 per interaction category, including 7 Candor and 8 synthetic pause clips. We selected clips using the AR outputs, aiming to keep assistant speech/silence proportions similar to the full benchmark. Appendix E.3 now explains this selection.

    Below, cc is frames advanced per backbone call. Revision (%) shows how often new user input triggers the next backbone call before all retained frames are used. Each clip's final interval is excluded from this percentage; cc includes all frames and calls.

    TaskClipsLISTEN cLISTEN revision (%)SPEAK cSPEAK revision (%)
    Pause (Candor)73.0765.43.2970.1
    Pause (Synthetic)82.9258.53.0664.6
    Backchannel152.5659.82.7767.4
    Turn taking152.0228.52.1934.1
    Interruption151.384.31.7010.0
    Overall601.9829.82.2637.6

    LISTEN/SPEAK denote DiffuPlex-LISTEN/DiffuPlex-SPEAK. This table is now in Appendix I.1, Table 19. The original Figure 7 analysis uses all 727 clips, with revision rates of 23.6%/32.1%.

    Multi-frame use and revision can occur in the same interval. Among non-final intervals in the subset, 23.8%/37.9% advance multiple frames. Figure 9 in Appendix I.3 already illustrates this: 3 of 12 retained frames are consumed before the remaining 9 are revised. Figures 8 and 11 show consecutive 16-frame advances and repeated one-frame advances, respectively; further audio examples are on the demo page.

  • Table 2 remeasured on all 727 clips.

    To check whether the subset affects the conclusions, we additionally measured the full benchmark. Times are cumulative seconds over 727 clips, not the 60 in Table 2; parentheses give speedups relative to RL-Seamless. Streaming wall is elapsed loop time; E2E GPU includes the codec; Core LM covers language-model computation.

    Modelc ↑Streaming wall ↓E2E GPU ↓Core LM ↓
    Moshi1.006424.3 (1.170×)6420.0 (1.170×)5526.0 (1.195×)
    PersonaPlex1.007500.8 (1.002×)7496.3 (1.002×)6604.7 (1.000×)
    RL-Seamless1.007514.1 (1.000×)7509.6 (1.000×)6605.8 (1.000×)
    DiffuPlex-LISTEN1.915314.3 (1.414×)5298.9 (1.417×)4192.3 (1.576×)
    DiffuPlex-SPEAK2.234755.5 (1.580×)4740.9 (1.584×)3684.9 (1.793×)
    ModelRelative Core LM time/wakeWake-frame p99 ↓Wakes >80ms ↓
    Moshi0.84×--
    PersonaPlex1.00×--
    RL-Seamless1.00×52.60ms0%
    DiffuPlex-LISTEN1.21×66.61ms0%
    DiffuPlex-SPEAK1.24×66.68ms0%

    Relative time/wake is normalized to RL-Seamless; p99 concerns frames invoking the backbone. The full benchmark also shows runtime savings, with all measured wakes below 80ms.

  • Does revision cancel the savings?

    On the timed 60-clip subset, intervals consuming multiple frames before revision have these mean costs:

    ModelFrames consumedProcessing timeEstimated AR time
    DiffuPlex-LISTEN4.8185.83ms224.69ms
    DiffuPlex-SPEAK4.6985.24ms219.19ms

    These are cumulative processing times across several frames, not single-frame latencies. They include the initial prediction, including subsequently discarded frames, and all frames consumed before the next wake. The AR estimate uses the same clip's mean AR frame time for the same number of frames. Later revision wakes are included in the total runtimes above.

    We also ran forced timing extremes over all 727 clips. Forcing revision at every check gives wall-clock speedups of 0.799×/0.786× for DiffuPlex-LISTEN/DiffuPlex-SPEAK, so immediate revision can be slower than AR. Disabling confidence filtering and revision gives 2.203×/4.601×; LISTEN still stops at non-PAD states. These are timing stress tests, not quality-preserving settings. Revision after several consumed frames can still save computation, while immediate revision can erase the benefit.

Thank you for prompting us to clarify the subset and test whether the timing benefits persist over the complete benchmark.

Q1. When does the backbone wake, and how is draft consistency assessed?

Thank you for asking us to clarify this procedure. User frames arrive every 80ms, but the backbone does not predict a new block at every arrival. At each wake, it predicts several future user and assistant frames, retains an initial assistant prefix for playback, and stores the user semantic-token distributions and assistant text-token probabilities needed for revision. It predicts a new block only when the retained frames run out or incoming user input triggers revision.

The check uses probabilities rather than an exact token match. When a user frame arrives, we look up the probability assigned to its semantic token in the stored user prediction. We combine this with a term based on the probabilities of the assistant's selected text token and its alternatives. Revision becomes more likely when the user input was unexpected and the assistant was uncertain about its response. No additional backbone pass is needed for this check. If the score reaches τrev\tau_{\mathrm{rev}}, the unplayed future is discarded and re-predicted at the next frame; received user input and played assistant audio remain unchanged. The rule and derivation were already provided in Section 3.3, Eq.4, and Appendix D.3. We have clarified what is stored and how the per-frame check differs from a backbone wake.

W1. What does "relatively coarse chunks" mean?

Thank you for pointing out this ambiguity. We intended to compare interaction updates with the 80ms frame interval (12.5Hz) used by Moshi and PersonaPlex. SyncLLM uses 160-240ms chunks (approximately 4.17-6.25Hz), while MiniCPM-o 4.5 reports selecting 1s chunks (1Hz). We have revised the Introduction to give these numbers and distinguish these chunk-based models from models whose user and assistant streams advance together at a finer frame-level granularity. Single-sequence serialization does not itself imply coarse updates.

W6 & Q6. Relationship to masked prediction, diffusion, and speculative decoding.

Thank you for encouraging us to clarify these connections. Masked prediction is shared by BERT-style objectives and masked diffusion: MDLM (Sahoo et al., 2024), for example, derives its objective as a mixture of masked-language-modeling losses. The specific construction we directly adopt is the clean-context training layout of Block Diffusion (Arriola et al., 2025, Section 3.2), already cited in Section 3.1. It processes clean and masked sequences together so that masked future blocks condition on the preceding realized interaction (Appendix C.3, Figure 4).

DiffuPlex adapts this layout to rolling full-duplex prediction. Masks reflect which future inputs are unavailable at each lookahead. Each wake predicts multiple future positions in one temporal-backbone pass, with causal attention and distillation from the frozen AR backbone. A confident prefix is retained, and future prediction repeats as new user input arrives. Related methods include causal masked prediction with confidence-based prefix commitment in FLUID (Ma et al., 2026), one-step masked drafting in TiDAR (Liu et al., 2025), and frozen-AR distillation with gated LoRA in Uno (Sahoo et al., 2026). Section 3.1 now explicitly identifies the adopted training layout, and Section 2 discusses these related methods.

Distillation describes how the predictor is trained; speculative decoding involves verifying proposed tokens during generation. DiffuPlex performs no AR verification or rejection sampling and does not claim exact preservation of the AR output distribution. Instead, Eq.4 checks predictions against newly arriving user input and revises only the unplayed future. This addresses the full-duplex setting, where future user speech is unavailable when the draft is generated.

W4. Clarifying Figure 2 and the role of stored predictions.

Thank you for these suggestions. We apologize that Figure 2 was difficult to follow. Panel(a) now labels the streams and codebooks, separates realized interaction from masked future inputs, and distinguishes the training-only paths. We also corrected an illustration-only masking error: assistant inputs are masked from h=2h=2, user codebook 0 (cb0) from h=3h=3, and user cb1-7 from h=4h=4, consistent with Appendix C.2-C.3, Eq.8, and Figure 4. The implementation and reported results are unchanged.

Panel(b) connects stored user predictions U1/U2U_1/U_2 to observed tokens X1/X2X_1/X_2 through schematic probability distributions, with the observed tokens highlighted. These probabilities contribute to the revision score together with the assistant-side term in Eq.4. In the example, observing X1X_1 gives S1<τrevS_1<\tau_{\mathrm{rev}}, so the retained draft continues without another backbone call. Observing X2X_2 gives S2≥τrevS_2\geq\tau_{\mathrm{rev}}, triggering re-masking of the unplayed future and prediction at the next frame. The figure also distinguishes the retained prefix, played audio, and unplayed future.

Reviewer_78XHOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DiffuPlex uses rolling masked diffusion to accelerate full-duplex inference by predicting future interaction frames and revising the unplayed future. Experiments mainly on PersonaPlex-RL-Seamless show useful speedups, with better quality preservation in LISTEN than in SPEAK.

Critical strengths
  • The prediction-and-revision approach is an interesting way to reduce sequential inference cost while staying responsive to incoming audio. It addresses a practical bottleneck in full-duplex systems.

  • Runtime measurements, human evaluation, and ablations support the method. LISTEN provides a useful speedup with little quality loss.

Critical weaknesses
  • Generalization is still limited. Appendix H.1 shows interaction degradation when transferring to the original PersonaPlex checkpoint, and there is no validation on a substantially different full-duplex dual-stream architecture.
  • SPEAK acceleration noticeably reduces speech naturalness. This limits the benefit during assistant speech, even though the LISTEN results remain useful.
Questions for authors

1、What explains the transfer degradation in Appendix H.1, and what model or training assumptions does DiffuPlex rely on?

2、Can existing logs or ablations clarify the SPEAK quality–speed trade-off and why its quality loss is larger than in LISTEN?

3 comments

Thank you for your careful reading and constructive feedback. Your questions prompted us to examine backbone transfer and speech naturalness more closely. Following reviewer fkjs’s helpful suggestion, we have re-evaluated all FDB v1.0 interruption-judge scores using gpt-4o-2024-08-06 with the official evaluation prompt and scoring procedure. All corresponding scores below use this judge. We address your weaknesses and questions together below. Changes in the revised manuscript are highlighted in red for ease of reference.

W1 & Q1. Transfer degradation and the conditions for generalization.

Thank you for highlighting the PersonaPlex results in Appendix H.1, Table 11 of the original submission. We agree that these results limit a broad generalization claim: applying the main recipe produced substantially longer interruption latencies, 1.277s for DiffuPlex-LISTEN and 1.347s for DiffuPlex-SPEAK, versus 0.218s for PersonaPlex AR.

Unlike our main backbone, RL-Seamless, PersonaPlex has not undergone interactivity-focused RL post-training. We hypothesized that differences in interaction training could affect how well speaking/listening decisions survive masked-future distillation. To preserve these decisions more strongly, we increased the weight of the three-way PAD/EPAD/lexical text-state KL term in Eq.1 from 1.0 to 8.0.

Below are the resulting PersonaPlex-7B-v1 results on all 727 FDB v1.0 clips, using τrev=−2.0\tau_{\mathrm{rev}}=-2.0. The PersonaPlex AR row uses the original Table 1 outputs, rescored with GPT-4o. Here, cc denotes frames advanced per backbone call; latencies are in seconds.

Modelc ↑Pause S TOR ↓Pause C TOR ↓BC TOR ↓BC FreqBC JSD ↓Turn TOR ↑Turn latency ↓Int TOR ↑Int GPT-4o ↑Int latency ↓
PersonaPlex AR1.000.3070.3430.2000.0580.8240.9410.2850.9304.5430.218
DiffuPlex-LISTEN1.620.4090.4540.2360.0880.7710.9580.2690.9304.5320.297
DiffuPlex-SPEAK1.790.3650.4490.2360.0940.7650.9580.1920.9414.6490.269

With these settings, the adapters reduce backbone calls while retaining broadly similar conversational behavior, with remaining differences in individual metrics and response latency. The updated results and adaptation settings are included in Appendix H.1, Table 12.

The reported PersonaPlex adapter checkpoint was reached after 263 steps, requiring approximately 2.5h on 4 RTX A6000 GPUs. For comparison, training the main-backbone adapter requires approximately 5h on 4 RTX A6000 GPUs. Only MASK embeddings and future-gated LoRA parameters are trained; the backbone and its original h=1h=1 path remain unchanged.

However, extending DiffuPlex to other full-duplex architectures is not always straightforward. The complete method currently assumes aligned user/assistant streams, predicted user-token probabilities for revision, and a PAD listening state for DiffuPlex-LISTEN retention. Eq.4 uses the probability assigned to a newly observed user semantic token by an earlier prediction. Models such as Raon-SpeechChat-9B encode incoming user audio but do not predict future user tokens, so this probability is unavailable. Extending DiffuPlex to such models would require a user-token predictor or a revision rule that works directly with newly received user input.

For reference, our training-free silence-retention baseline on Raon follows the same call-skipping idea as the deferred-execution baselines in Section 4.3. It yields a 1.20× step speedup, but changes pause and backchannel behavior. This experiment does not use DiffuPlex’s learned future predictor or user-aware revision and therefore does not establish complete transfer to that architecture. Developing revision without explicit future-user prediction is a valuable next step toward broader applicability.

We sincerely thank you for helping us distinguish the demonstrated backbone adaptation from the further development required for different architectures. We have clarified the adaptation settings in Appendix H.1 and the architectural limitation in Appendix A.

W2 & Q2. Why DiffuPlex-SPEAK reduces speech naturalness more than DiffuPlex-LISTEN.

Thank you for encouraging us to explain this difference more clearly. In Table 1's offline human evaluation, NMOS remains similar for the backbone and DiffuPlex-LISTEN (3.93±0.12 and 3.92±0.11), but decreases to 3.54±0.12 for DiffuPlex-SPEAK.

Each 80ms assistant audio frame contains a semantic token, codebook 0 (cb0), and seven acoustic tokens, cb1-7. In ordinary AR decoding, cb0 is generated first, and cb1-7 for that same audio frame are generated one temporal step later. At that later step, cb0 is already an input to the backbone, helping it predict the corresponding acoustic tokens.

DiffuPlex preserves this native delay, but computes multiple future temporal representations together, with future assistant inputs masked. The generated cb0 for a future audio frame is therefore not fed into the temporal representation used to predict that frame’s cb1-7. Future acoustic predictions lack the same-frame cb0 conditioning available during ordinary AR decoding. DiffuPlex-LISTEN retains future frames only during predicted listening, while speech is generated through the frozen h=1h=1 path. DiffuPlex-SPEAK also plays speech predicted at h≥2h\geq2, so these acoustic errors can affect the speech listeners hear.

The original submission examined this explanation through an autoregressive replay (Section 5.5 and Appendix I.2). Starting from each DiffuPlex-SPEAK output, we kept its generated text and cb0 tokens fixed at every frame. We then evaluated the frozen backbone once per frame, following its original AR decoding procedure, and regenerated only cb1-7 with the unchanged depth transformer. This restores ordinary AR audio-token conditioning while preserving DiffuPlex-SPEAK’s text and semantic-token sequence. After decoding the regenerated tokens into audio, UTMOSv2, an automatic speech-quality predictor, increases from 2.646 to 2.894, compared with 2.919 for fully AR generation in the same scoring run. Recovering most of this score gap supports the acoustic-generation path, including the loss of same-frame cb0 conditioning, as a major contributor to the naturalness loss.

We additionally scored the stored outputs from the confidence-threshold ablation (Appendix H.4, Table 15). Confidence filtering uses the assistant text-token confidence to determine how many future frames are retained at each wake. A higher threshold accepts fewer future frames and brings the backbone back sooner to its original next-frame prediction path (h=1h=1). A lower threshold permits longer drafts and fewer wakes, increasing reliance on masked future acoustic predictions. User-aware revision remains enabled throughout these experiments.

We evaluated the stored assistant audio from FDB v1.0 interruption examples with UTMOSv2 and silence removal enabled:

ConfigurationFrames per wake, c ↑UTMOSv2 ↑
RL-Seamless1.002.936
DiffuPlex-SPEAK, τconf=0.90\tau_{\mathrm{conf}}=0.902.072.720
DiffuPlex-SPEAK, default τconf=0.80\tau_{\mathrm{conf}}=0.802.232.632
DiffuPlex-SPEAK, no confidence filtering2.802.210

Raising the confidence threshold reduces frames consumed per wake while improving predicted naturalness. Removing confidence filtering also lowers GPT-4o interruption appropriateness from 4.648 to 3.470.

These results connect the larger naturalness loss in DiffuPlex-SPEAK to its use of future speech predictions and show that more conservative retention reduces this loss. We retain DiffuPlex-LISTEN as the primary quality-preserving setting and present DiffuPlex-SPEAK as an optional policy with an explicit naturalness cost. We sincerely thank you for helping us explain both the acoustic mechanism and the available quality-computation choices more clearly.


We hope these explanations and additional results address your questions. We welcome any further questions or discussion, and thank you again for your time and careful consideration.

Thank you again for your questions about transfer and speech naturalness. The updated PersonaPlex adaptation is in Appendix H.1, Table 12. Our W1/Q1 response explains why we strengthened text-state distillation to preserve speaking/listening decisions. The adaptation settings are in Appendix H.1, and Appendix A states the architectural limitations. We marked these manuscript changes in red.

For speech naturalness, our W2/Q2 response explains the original autoregressive replay. Keeping DiffuPlex-SPEAK's text and semantic tokens fixed while regenerating the acoustic tokens autoregressively raises UTMOSv2 from 2.646 to 2.894, compared with 2.919 for fully AR generation in the same scoring run. This recovery supports the loss of ordinary AR conditioning, including the same-frame semantic input to the temporal backbone, as a contributor to the naturalness loss. The additional confidence-threshold results show that more conservative retention, which returns more often to AR prediction, reduces the measured quality loss.

We would appreciate your thoughts on whether these results explain the transfer behavior and the larger naturalness loss in DiffuPlex-SPEAK. We would be happy to discuss any points that still need clarification. Thank you for your time and helpful suggestions.

Reviewer_Fxh4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DiffuPlex, a rolling masked-future prediction framework to accelerate full-duplex spoken dialogue models. It predicts multiple future interaction frames in a single backbone invocation and uses confidence-based retention and user-aware revision to reduce sequential computation. While the method achieves measurable inference speedups, more aggressive prediction degrades speech naturalness.

Critical strengths
  1. The paper addresses a relevant problem in real-time spoken dialogue and presents a structured prediction and revision mechanism.

  2. The experiments evaluate runtime efficiency, conversational behavior, and speech quality.

  3. The ablation studies provide insights into the trade-offs associated with future prediction and revision.

Critical weaknesses
  1. Unconvincing efficiency–quality trade-off: The claimed acceleration comes at the expense of response quality and, in some scenarios, responsiveness. Timely and adaptive responses are fundamental properties of full-duplex dialogue, rather than optional features that should be sacrificed for computational efficiency. The paper does not sufficiently justify why this trade-off is acceptable or demonstrate that the acceleration brings corresponding practical benefits.

  2. Limited generalizability and additional training cost: The method requires additional training of MASK embeddings and LoRA parameters, while the main evaluation focuses on a single backbone. Its transferability to other full-duplex architectures and the associated training costs remain unclear.

  3. Insufficient evidence of system-level efficiency gains: Although the paper reports single-stream runtime and accumulated GPU computation time, it does not investigate GPU utilization, energy consumption, or whether the method enables more concurrent dialogue sessions on the same hardware. Therefore, its practical benefits in terms of overall resource efficiency remain insufficiently established.

Questions for authors
  1. What practical benefits justify sacrificing responsiveness and speech quality for reduced inference computation?

  2. Can the method generalize to different full-duplex backbones without substantial additional training?

  3. Does DiffuPlex improve system-level efficiency, such as concurrent dialogue throughput or resource utilization, beyond single-stream runtime speedups?

6 comments

Thank you for this thoughtful review. Your questions prompted us to reassess how we present and substantiate the practical benefits of DiffuPlex. Following reviewer fkjs’s helpful suggestion, we have re-evaluated all FDB v1.0 interruption-judge scores using gpt-4o-2024-08-06 with the official evaluation prompt and scoring procedure. All corresponding scores below use this judge. We address your weaknesses and questions together and describe the corresponding manuscript revisions. Changes in the revised manuscript are highlighted in red for ease of reference.

W1 & Q1. Practical benefits versus responsiveness and quality.

Thank you for raising this concern. We agree that timely, adaptive responses are fundamental to full-duplex dialogue. Your question helped us clarify the distinction between our two policies: DiffuPlex-LISTEN consumes multiple future frames only during predicted sustained listening, whereas DiffuPlex-SPEAK can also consume predicted assistant speech. We emphasize DiffuPlex-LISTEN as the primary setting and present DiffuPlex-SPEAK as an optional, more aggressive policy with an explicit speech-naturalness cost.

For DiffuPlex-LISTEN, mean turn-taking latency increases from 124ms to 173ms, and interruption response latency from 281ms to 329ms. We acknowledge these increases. Other FDB v1.0 metrics show a different pattern: synthetic-pause TOR decreases from 0.299 to 0.277, backchannel TOR from 0.218 to 0.109, and GPT-4o interruption scores remain similar (4.578 vs 4.591). We report these results to characterize the effects of acceleration, without claiming improved turn-taking or treating an individual metric as evidence of better interaction quality. Our objective is to reduce computation while largely retaining the backbone’s existing behavior.

The paired tests in Appendix G.1, Table 6, did not detect statistically significant differences across the evaluated behavioral metrics after Holm correction. This does not establish equivalence or tell us how the observed changes affect listeners’ experience. To assess the interaction as a whole, we also measured DMOS over the complete exchange, including whether the assistant yields to an interruption and resumes at appropriate times. DiffuPlex-LISTEN achieves a DMOS of 4.08±0.13, compared with 4.09±0.14 for the backbone. Speech naturalness is also rated similarly: NMOS is 3.92±0.11 versus 3.93±0.12. These ratings provide complementary evidence that perceived conversational quality and speech naturalness remain close to the backbone in the evaluated interactions.

Alongside these similar human ratings, DiffuPlex-LISTEN provides measurable computational savings: accumulated GPU execution time for the complete inference pipeline decreases from 745.3s to 511.1s on the fixed 60-clip subset, a 31.4% reduction and a 1.46× speedup. Additional real-time experiments show lower GPU utilization and energy consumption, together with smaller increases in response delay when concurrent conversations share a GPU. We discuss these practical benefits in W3/Q3 below. We appreciate your guidance in helping us distinguish the observed behavioral changes, perceived quality, and deployment benefits more clearly.

W3 & Q3. System-level efficiency and concurrent conversations.

Thank you for suggesting these measurements. Your question prompted us to examine whether fewer backbone calls translate into practical resource savings and better support for concurrent conversations. We conducted additional experiments with user frames arriving every 80ms, measuring GPU utilization, power, energy, and response delays.

  • GPU utilization and energy consumption.

    With one session on an RTX A6000, using the fixed 60-clip runtime subset, GPU utilization decreases from 58.0% for RL-Seamless to 38.0% for DiffuPlex-LISTEN and 34.4% for DiffuPlex-SPEAK. Mean GPU power decreases from 175.9W to 147.6W and 141.0W. GPU energy consumption decreases by 16.1% and 19.9%, with 95% confidence intervals of [12.6,19.4]% and [17.4,22.5]%, respectively. These measurements demonstrate resource savings even when a single RL-Seamless session already meets the 80ms frame deadline.

  • Concurrent conversations on one GPU.

    We evaluated all 727 FDB v1.0 clips with 3 concurrent sessions on an RTX PRO 6000 Blackwell Max-Q (96GB), each using its own model instance on the shared GPU, with one run per setting. If a frame’s output is not ready within its 80ms interval, the resulting audio is buffered for later playback. The same policy is applied to all systems, and latency and GPT-4o scores are measured on the same playback audio using the official FDB scoring procedure.

    We report the official Smooth Turn Taking metrics (TOR and latency) and User Interruption metrics (TOR, Judge, and latency):

    ModelTurn TOR ↑Turn latency ↓Int TOR ↑Int Judge ↑Int latency ↓
    RL-Seamless0.8070.894s0.9854.5332.233s
    DiffuPlex-LISTEN0.9830.191s0.9904.5511.524s
    DiffuPlex-SPEAK0.9920.203s1.0004.6350.914s

    TOR measures how often the assistant takes a conversational turn in the corresponding task. Latency measures the interval from the end of the user’s utterance to the assistant’s first response word. Judge measures interruption-response appropriateness using GPT-4o.

    Increasing from 1 to 3 sessions raises turn-taking latency from 0.101s to 0.894s for RL-Seamless, from 0.144s to 0.191s for DiffuPlex-LISTEN, and from 0.156s to 0.203s for DiffuPlex-SPEAK. The increase is therefore 793ms for RL-Seamless and 47ms for either DiffuPlex policy. Both policies also achieve GPT-4o interruption scores similar to RL-Seamless at the same load.

    At 3 sessions, GPU energy per conversation hour decreases from 84.7Wh for RL-Seamless to 57.9Wh for DiffuPlex-LISTEN and 52.9Wh for DiffuPlex-SPEAK, reductions of 31.6% and 37.5%, respectively.

    We also tested 2 concurrent sessions on the A6000. Turn-taking latency is 0.832s for RL-Seamless, 0.226s for DiffuPlex-LISTEN, and 0.193s for DiffuPlex-SPEAK; corresponding GPT-4o interruption scores are 4.518, 4.581, and 4.618.

These results connect the computational savings to lower GPU resource consumption and more stable response times at the tested concurrent loads. Section 5.2 now reports the single-session utilization and energy savings, together with the concurrent-session setup and complete FDB v1.0 automatic metrics in Table 3. We appreciate your guidance in helping us substantiate these system-level benefits beyond single-session runtime speedups.

W2 & Q2. Generalization and additional training cost.

Thank you for encouraging us to examine how the method transfers and what additional training it requires. We applied DiffuPlex to PersonaPlex-7B-v1 as an additional backbone. In the original submission (Appendix H.1, Table 11), applying the main recipe to PersonaPlex changed its pause handling and interruption behavior substantially. This prompted us to reconsider which aspects of the teacher’s behavior need stronger preservation during adaptation.

Our main backbone, RL-Seamless, is post-trained for interactive behavior. We hypothesized that differences in interaction training could affect how well speaking/listening decisions survive masked-future distillation. This motivated stronger regularization toward the AR teacher: we increased the weight of the three-way PAD/EPAD/lexical text-state KL term in Eq.1 from 1.0 to 8.0. The backbone remains frozen; only MASK embeddings and future-gated LoRA parameters are trained, preserving the original h=1h=1 path.

The resulting PersonaPlex adapter was evaluated with τrev=−2.0\tau_{\mathrm{rev}}=-2.0 over all 727 FDB v1.0 clips. The PersonaPlex AR row uses the original Table 1 outputs, rescored with GPT-4o. Here, cc denotes frames advanced per backbone call, and all latencies are in seconds.

Modelc ↑Pause S TOR ↓Pause C TOR ↓BC TOR ↓BC FreqBC JSD ↓Turn TOR ↑Turn latency ↓Int TOR ↑Int GPT-4o ↑Int latency ↓
PersonaPlex AR1.000.3070.3430.2000.0580.8240.9410.2850.9304.5430.218
DiffuPlex-LISTEN1.620.4090.4540.2360.0880.7710.9580.2690.9304.5320.297
DiffuPlex-SPEAK1.790.3650.4490.2360.0940.7650.9580.1920.9414.6490.269

These results support adaptation to another backbone, reducing backbone calls while retaining broadly similar conversational behavior, with remaining differences in individual metrics and response latency. The updated results and adaptation settings are included in Appendix H.1, Table 12.

This adaptation requires additional parameter-efficient training. The reported PersonaPlex adapter checkpoint was reached after 263 steps, requiring approximately 2.5h on 4 RTX A6000 GPUs. For comparison, training the main-backbone (RL-Seamless) adapter requires approximately 5h on 4 RTX A6000 GPUs.

However, extending DiffuPlex to other full-duplex architectures is not always straightforward. The current method assumes aligned user/assistant frames and uses predicted user-token probabilities for revision. Specifically, Eq.4 checks the probability assigned to a newly observed user semantic token by an earlier prediction. Models such as Raon-SpeechChat-9B encode incoming user audio but do not predict future user tokens, so this probability is unavailable. Extending DiffuPlex to such models would require a user-token predictor or a revision rule that works directly with newly received user input.

For reference, we also tested training-free silence retention on Raon, skipping LLM calls while retaining silence frames. This follows the same call-skipping idea as the deferred-execution baselines in Section 4.3, allowing us to measure its computational savings on this architecture. On an RTX A6000, the official decoding loop takes 229.0ms per 80ms interaction frame; this baseline reduces it to 191.2ms, a 1.20× step speedup. As with those baselines, these savings accompany changes in pause and backchannel behavior. This experiment serves as a simple computational baseline and does not use DiffuPlex’s learned future predictor or user-aware revision. Developing revision without explicit future-user prediction is a valuable next step toward extending the complete method to such architectures. We have clarified this architectural limitation in Appendix A, and sincerely thank you for motivating this investigation.


We hope these explanations and additional results address your questions. We welcome any further questions or discussion, and thank you again for your time and careful consideration. If our responses resolve the issues underlying your evaluation, we would be grateful if you would consider revisiting your assessment and score.

Comments on the paper

1 comment

Dear Reviewers, Area Chair, and Program Chairs,

Thank you for taking the time to read our responses and discuss the additional experiments. We appreciated hearing which points our responses clarified and which concerns remained. Your questions prompted us to test DiffuPlex with several conversations sharing a GPU and to explain more clearly when the method is useful. We would like to share our perspective on the remaining concerns about responsiveness and practical benefit, and explain how your feedback shaped the revised manuscript.

1. Does DiffuPlex sacrifice responsiveness to reduce computation?

With RL-Seamless as the backbone, DiffuPlex-LISTEN had 49ms higher mean turn-taking latency and 48ms higher mean interruption latency than the backbone in our original Full-Duplex-Bench v1.0 evaluation. The other interaction metrics showed a mixed pattern, with some numerically favoring LISTEN and others favoring RL-Seamless. None of the 20 paired behavioral comparisons, including those for latency, reached statistical significance after Holm correction (Appendix G.1, Table 6). With PersonaPlex as the backbone, the same evaluation protocol and random seed yielded shorter mean turn-taking latencies for both policies. The mean was 0.285s for the backbone, 0.269s for LISTEN, and 0.192s for SPEAK.

We also asked listeners to rate complete interruption exchanges. This evaluation takes timing into account, including whether the assistant yields to the user and resumes speaking at an appropriate time. Listeners gave LISTEN and RL-Seamless similar overall conversational-quality ratings (DMOS of 4.08 and 4.09, respectively). Considering the paired comparisons, the results on both backbones, and the listener ratings together, we believe that the roughly 50ms increase observed with RL-Seamless is insufficient evidence that LISTEN makes the assistant less responsive overall.

2. What do we gain from fewer backbone calls if each call takes longer?

A similar trade-off arises in diffusion language models that predict multiple positions together and in speculative decoding, where the target model checks several proposed tokens in one call. For a given backbone, processing several positions can take longer than a single autoregressive step and still reduce total execution time if fewer sequential calls are needed. To our knowledge, DiffuPlex is the first method for full-duplex spoken dialogue to jointly predict future user and assistant frames in one backbone call and use incoming user speech to decide whether to revise the unplayed assistant frames.

  • Single session. Across all 727 FDB v1.0 clips, LISTEN achieved a 1.58× Core LM speedup over RL-Seamless. We also measured GPU resource use with input frames arriving in real time. On the 60-clip runtime subset, LISTEN reduced mean GPU utilization on an RTX A6000 from 58% to 38%, mean GPU power from 175.9W to 147.6W, and GPU energy use by 16.1%. Serving the same clips therefore required less energy even when RL-Seamless already met the 80ms frame deadline.

  • Concurrent sessions. In a deployed dialogue service, several users may hold separate conversations at the same time on a shared GPU. With three sessions on a single PRO 6000 Max-Q, mean turn-taking latency was 0.191s for LISTEN and 0.894s for RL-Seamless, a difference of 703ms. With two sessions on an A6000, the corresponding latencies were 0.226s and 0.832s, a difference of 606ms. At both tested loads, LISTEN responded more than seven 80ms interaction frames earlier on average. We evaluated all 727 clips using the same playback policy for both systems, with one run per setting (Section 5.2, Table 3).

3. How did the discussion change the paper?

We marked the changes made in response to the initial reviews in red. These include the updated PersonaPlex results, the GPU and concurrent-session measurements in Section 5.2, and clarifications of the evaluation protocol and revision procedure. The later changes, marked in blue, address the remaining questions about responsiveness and practical benefit. Section 5.1 now discusses the paired behavioral comparisons alongside the human ratings, including the lower speech-naturalness ratings for SPEAK. We also renamed Section 5.2 to "Runtime and Resource Efficiency" and revised the Abstract, Introduction, and Conclusion to present resource efficiency as the main contribution. The revised main text fits within 10 pages.


If any questions remain or there is a point we have overlooked, we would welcome the opportunity to discuss it in the time remaining. Thank you again for reading our responses so carefully and for the time you have already given to the paper. Your feedback shaped both the additional experiments and how we present their results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.