acceptodds
Under review as a conference paper at ICLR 2027

DuplexOmni: Real-Time Listening, Seeing, Thinking, and Speaking for Full-Duplex Interaction

Abstract

Human interaction is continuous, multimodal, and full-duplex by nature. Although recent omni models have made substantial progress in unified speech, vision, and text modeling, combining seamless real-time interaction with complex reasoning and tool use remains challenging. We present DuplexOmni, a method for real-time multimodal full-duplex interaction. DuplexOmni separates model capability into an interaction layer and a thinking layer, which collaborate asynchronously in parallel. The interaction layer is implemented by the DuplexOmni model, an end-to-end system that processes streaming audio and video inputs while generating text and speech responses in real time. The thinking layer is a pluggable module that provides complex reasoning and tool-use capabilities. To support this method, we further develop a Writer-Director pipeline for constructing continuous-interaction training data. Experiments show that DuplexOmni achieves strong full-duplex interaction performance on public benchmarks while maintaining low response latency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Interactive Omni Model, Duplex Omni Model, Omni model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_x2rHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper decouples the streaming audio-visual interaction model from the asynchronous reasoning and tool-use layers. A Writer-Director pipeline was employed to construct timestamped training dialogues, encompassing scenarios such as interruptions and latency in reasoning feedback. Experimental results demonstrate the model's superior performance in turn-taking and conversational reasoning, while ablation studies further examine the contribution and substitutability of the reasoning layer.

Critical strengths
  1. DuplexOmni achieves 72.6% ToR on Full DuplexBench, compared with 36.3% for the strongest listed baseline, with 0.506 s response latency. The interaction-only variant retains 65.2%, supporting the value of the trained interaction model independently of the external reasoner.
  2. The Writer-Director pipeline provides a concrete approach to constructing continuous-interaction data. Its explicit representation of interruptions, shared silence, reasoning requests, and delayed feedback addresses an important training challenge for asynchronous voice systems.
Critical weaknesses

The central claim of effective reasoning and tool coordination during evolving interaction is not directly evaluated. Sections 4.4–4.5 rely on separate turn-taking and spoken-question benchmarks, while appendix contains training examples rather than held-out coordination results. First-audio-packet latency also does not measure useful task completion. A focused evaluation of interrupted reasoning or tool tasks could resolve this concern during rebuttal if suitable results already exist.

Questions for authors

Please report whether the system correctly incorporates changed user conditions during background reasoning or tool execution, including final-task correctness and time to a useful answer.

Reviewer_nV7nOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DuplexOmni combines a streaming audiovisual interaction model with a pluggable asynchronous thinking layer. Its main contributions are time-sliced Thinker–Talker inference, an interaction–thinking coordination protocol, and a Writer–Director pipeline for synthetic training data. Evaluation covers full-duplex turn-taking, spoken reasoning, audiovisual understanding, transcription, latency, and thinking-layer ablations.

Critical strengths
  1. Substantial engineering contribution. The system integrates continuous audiovisual perception, speech generation, and background reasoning. The released code, weights, and data-construction pipeline provide useful resources for further research.
  2. Promising empirical performance. Strong turn-taking and spoken-reasoning results demonstrate the practical effectiveness of the integrated system, while the ablations establish the benefit of external reasoning assistance.
Critical weaknesses
  1. Insufficiently established methodological novelty. MoshiRAG (ICML 2026) already combines a full-duplex frontend with model-triggered asynchronous external assistance and result reintegration, including external LLM assistance for mathematical reasoning. Mind-Paced Speaking and OmniFlatten provide precedents for dual-brain speech generation and synthetic duplex data. Although DuplexOmni adds audiovisual interaction and explicit reasoning control, it does not convincingly establish a sufficiently substantial methodological advance beyond this integration.
  2. Validation does not isolate the claimed advance. The ablations primarily demonstrate the benefit of a stronger external thinking model, rather than the proposed asynchronous coordination, cancellation/restarting behavior, or Writer–Director supervision. There is no controlled comparison against a simpler coordination strategy using the same components. First-audio latency also does not establish timely delivery of substantive answers, and interaction naturalness lacks human evaluation.
Questions for authors
  1. Clarification: What specific methodological advance distinguishes DuplexOmni from MoshiRAG and existing dual-brain systems? How do the concurrent Gander and Realtime-Venus systems further delimit that contribution?
  2. Evidence clarification: What evidence demonstrates the benefit of your coordination protocol and Writer–Director supervision when the backbone, thinking model, and training budget are controlled, particularly under user revisions and delayed background results?
Reviewer_AkUfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a method named DuplexOmni for real-time multimodal full-duplex interaction. The method separates model capability into an interaction layer and a thinking layer, and the two layers collaborate asynchronously in parallel. The interaction layer uses a Thinker-Talker architecture to process streaming audio and video inputs, and generate text and speech responses in real time. The thinking layer provides complex reasoning and tool-use capabilities. The authors also build a Writer-Director data pipeline to construct continuous interaction training data with explicit timing of turn-taking and thinking events. Experimental results show that the proposed method has strong full-duplex interaction performance with low response latency.

Critical strengths
  1. The paper separates the interaction layer from the thinking layer. This design allows real-time dialogue and background reasoning to proceed asynchronously. This design also addresses an important limitation of existing omni models, where reasoning and tool use can interrupt real-time interaction. The interaction layer can also incorporate streaming thinking results and handle user interruptions.

  2. Experimental result show that the proposed method achieves 72.6% ToR on Full DuplexBench, substantially outperforming the evaluated baselines. For the latency, it maintains a good response latency of 0.506 seconds. This demonstrates the effectiveness of the proposed method for real-time full-duplex interaction.

  3. The Writer-Director pipeline converts turn-based dialogues into continuous-interaction training data with explicit temporal control signals. This provides a practical way to construct training data for full-duplex interaction.

Critical weaknesses
  1. The main contribution is to allow real-time interaction and complex reasoning to run in parallel. In the experiments, the paper do not directly evaluate how well the model handles long reasoning processes while maintaining continuous interaction. The reported benchmarks mainly measure interaction quality and audio understanding. Is there any direct way to assess the effectiveness of the proposed asynchronous collaboration mechanism? I'm confused how well this collaboration works.

  2. In the paper, there are no quantitative results are reported for tool-use tasks. I'm confused how accurately and efficiently DuplexOmni handles tool calls, delayed tool results, and interruptions during tool execution. Could the give some discussions about it?

  3. DuplexOmni achieves only 53.8% on Daily-Omni, substantially below several evaluated omni models. I have concerns about the general multimodal capability and practical effectiveness of the proposed method. Could the authors give more discussions about it? Are there any data distribution gap between training and test?

Questions for authors
  1. Could the authors provide some quantitative results on tasks requiring tool calls or long reasoning?

  2. DuplexOmni achieves 53.8% on Daily-Omni, compared with 80.2% for MiniCPM-o 4.5 and 76.2% for Qwen3-Omni-Realtime-Flash. What causes this performance gap? Is it related to the time-sliced architecture, the training data, or the interaction with the thinking layer?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Discussion and Revisions

We thank the AC and the three reviewers for their time and constructive feedback. We are glad that, during the discussion, all three reviewers indicated that their main concerns have been addressed. Below we summarize the main results added during the rebuttal and where the changes appear in the revised PDF.

To assess coordination between the interaction layer and the thinking layer on held-out data (AkUf, nV7n, x2rH), we evaluated DuplexOmni on spoken GSM8K across 400 English and 400 Chinese test problems, excluding those that appear verbatim in our released training data. While the thinking layer works, the user changes a number or adds a missing condition. DuplexOmni reaches 85.0–92.2% final accuracy when the thinking layer gets the exact text of what the user said, and 77.6–87.3% when it gets only the speech. At most 1.8% of final answers keep the outdated value, and the correct answer is spoken a median of 8.7–11.8 s after the user stops speaking.

We also ran controlled comparisons with the same backbone, trained weights, and thinking layer (nV7n). Removing cancel/restart lowers accuracy in all eight cells by 6.9–26.1 points. With the thinking layer getting only the speech, single injection of the thinking result reduces accuracy in all four cells by 4.1–10.5 points. When thinking results arrive late and in pieces, DuplexOmni is 1.1–26.8 points more accurate than single injection and speaks the correct answer 3.5–7.6 s earlier.

On tool use and continuous interaction (AkUf, x2rH), the interaction layer relays results that must be spoken verbatim with 96.5% (Chinese) / 95.3% (English) field-level accuracy, and 95–96% of interjections during thinking are answered within 2 s.

Turning to Daily-Omni (AkUf), evaluating under the same streaming conditions brings DuplexOmni's base model down from 70.9% offline to 55.9%, and MiniCPM-o 4.5 from 80.2% (reported offline) to 60.2% in its official full-duplex mode. Time slicing alone costs the base model 1.6 points.

Revisions in the PDF

Table 1 caption and Section 4.1: the setting of each Daily-Omni number is stated.

Appendix J: the held-out evaluation above (Tables 9–11), including the controlled comparisons, delayed results, interjections, and tool-result relay.

Appendix E: the Daily-Omni comparison under offline and streaming settings (Table 4).

Appendix B: discussion of MoshiRAG and the concurrent Gander and Realtime-Venus.

Appendix H: labeled as training examples.

Appendix A: a human evaluation of interaction naturalness and a direct ablation of Writer–Director supervision are listed as future work.

We are grateful to the reviewers for helping us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.