acceptodds
Under review as a conference paper at ICLR 2027

StreamPro-Duplex: A Full-Duplex Multimodal Agent Framework for Always-On Streaming Interaction

Abstract

Always-on streaming interaction requires assistants to continuously participate in open-ended interactions and to determine not only when to engage and what to say, but also how to interact with the user and which intent to serve as user multiple needs emerge, coexist, and evolve over time. Existing streaming systems have advanced the first two capabilities but remain limited in the latter two, largely following an Isolated Interaction paradigm that segments interaction into separate turns and handles user intents independently. Such isolation hinders natural full-duplex communication and makes it difficult to track multiple ongoing user needs across interleaved interactions. To address these limitations, we propose StreamPro-Duplex, a full-duplex multimodal agent framework for always-on streaming interaction. For how to interact, we decouple streaming understanding from interaction control and train a plug-and-play Duplex Controller that continuously determines and executes appropriate full-duplex actions. For which intent to serve, we design a persistent intent management mechanism that maintains evolving intents across interactions and coordinates concurrently triggered intents according to their priorities. We further train a unified Intent Monitor that jointly monitors multiple proactive intents over the shared streaming context in a single inference, avoiding repeated model execution for each intent. To support training, we construct StreamInteract-138K, a large-scale dataset covering representative always-on streaming interaction scenarios. Experiments demonstrate that StreamPro-Duplex achieves strong performance across full-duplex interaction, real-time streaming, and proactive streaming tasks, validating its effectiveness for always-on streaming interaction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Full Duplex, Streaming Video

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WpJpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StreamPro-Duplex, a full-duplex multimodal agent framework for always-on streaming interaction. The system decouples real-time interaction control from understanding through a plug-and-play Duplex Controller, introduces persistent multi-intent management, and trains a unified Intent Monitor that jointly monitors multiple proactive intents. The work further constructs StreamInteract-138K, covering full-duplex and multi-intent interaction scenarios.

Critical strengths
  • The paper presents a practical framework for integrating full-duplex control with persistent multi-intent streaming interaction. The separation between the Duplex Controller, intent management, and downstream monitoring/generation produces a relatively clean modular system.
  • The paper constructs StreamInteract-138K, a dataset covers diverse full-duplex interaction behaviors as well as concurrent and evolving proactive intents, providing large-scale supervision for aspects of always-on interaction.
Critical weaknesses
  • The paper is motivated by always-on interaction, where multiple real-time and proactive intents may coexist, persist, and evolve, and addresses this challenge through decoupled full-duplex control and persistent multi-intent management. However, the Duplex Controller appears conceptually close to turn-taking works [1] while the Intent Manager resembles persistent task-memory or task-state management in recent agent systems [2, 3, 4]. Given these conceptual overlaps, the paper could more clearly discuss in the Related Work what fundamentally distinguishes the design.
  • The current experiments do not clearly disentangle the contributions of the backbone models, task-specific training, and the proposed system design. For example, in Table 1, the Duplex Controller is initialized from Qwen3-Omni-30B-A3B, but the unadapted backbone is not included as a baseline on Full-Duplex-Bench. Table 2 further shows that the adapted controller has a higher WER than the original Qwen3-Omni backbone. Similarly, the strong “training-free” results in Table 4 use Qwen3.6-35B-A3B as the backbone for the Memory Manager and Real-Time Generator, while the compared methods use heterogeneous backbones and training regimes. Without a same-backbone comparison or ablations, it remains unclear whether the gains primarily arise from the strong backbone or from the proposed framework.
  • The experiments do not directly validate some of the paper’s central multi-intent claims. The claimed efficiency advantage of the unified Intent Monitor is not evaluated against a per-intent monitoring baseline as the number of active intents increases. There is also limited quantitative evaluation of persistent intent management under nested or heavily interleaved interactions.

[1] A Survey of Recent Advances on Turn-taking Modeling in Spoken Dialogue Systems, ACL 2025

[2] IPIBench: Evaluating Interactive Proactive Intelligence of MLLMs under Continuous Streams, to be accepted NIPS 2026

[3] Proactive Assistant Dialogue Generation from Streaming Egocentric Videos, EMNLP 2025

[4] Memento: Toward an All-Day Proactive Assistant for Ultra-Long Streaming Video, ICLR2026

Questions for authors
  1. Could the authors clarify the key methodological distinctions between the proposed Duplex Controller / Intent Manager and closely related full-duplex control and persistent task-state management paradigms? In particular, which capabilities are fundamentally enabled by the proposed design and how they influence always-on tasks?
  2. Could the authors provide more controlled experiments to disentangle the contributions of the backbone models, task-specific training, and the proposed framework?
  3. Could the authors provide additional experimental evidence or analysis to more directly substantiate the central multi-intent claims, particularly regarding persistent intent management and the effectiveness and efficiency of unified multi-intent monitoring under complex interleaved interactions?
  4. Could the authors provide the complete category-level results for MOSS-VL-Realtime and Streamo in Table 4, if available, or clarify why some results are not reported?
2 comments

Dear Reviewer WpJp,

Thank you again for your valuable feedback. We would like to kindly check whether our previous response has addressed your concerns, or if there are any remaining questions or issues that we could further clarify.

We would greatly appreciate any additional feedback.

Thanks for the responses. The additional concurrent-monitoring experiments with Intent Monitor address some of my concerns about its effectiveness. However, my main concerns remain the methodological distinctions and the attribution of the reported gains:

  1. Methodological distinctions. Decoupling control from generation is a common idea in turn-taking. Might the novelty instead lie in more specific differences in state representation, decision-making, and training methods? The characterization of IPI-Agent may also need clarification. It already supports task-state management through Proactive Memory, Interaction Memory, and an Intent Router, and processes different tasks independently and in parallel.

  2. Attribution of gains remains unclear. For Table 4, the additional matched-backbone comparison quantifies positive gains of +2.1/+1.3 from the framework, but also highlights that the backbone itself is already strong. The claims about the framework's advantages in Table 4 caption and main text should therefore be more carefully qualified. A similar attribution issue arises for Intent Monitor: Table 3 compares models such as Qwen3-VL-8b, but does not include its original Qwen3.6-35B-A3B backbone. This makes it difficult to disentangle the contributions of the backbone, training, and framework.

  3. Could you please list the per-category scores for MOSS-VL-Realtime?

Reviewer_BRBcOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StreamPro-Duplex, a full-duplex multimodal agent framework for always-on streaming interaction that aims to move beyond isolated interactions by supporting real-time interaction control and persistent multi-intent management.

Critical strengths

S1. Well-Motivated Problem Formulation and Modular Framework Design

The paper addresses an important challenge in always-on streaming interaction by jointly considering full-duplex interaction control and persistent multi-intent management. This modular design provides a practical approach to coordinating real-time interactions and evolving user intents while leveraging existing backbone models.

S2. Comprehensive Dataset Construction and Promising Experimental Results

The authors construct StreamInteract-138K, covering diverse full-duplex behaviors and multi-intent interaction scenarios. The experiments also demonstrate promising results across several benchmarks.

Critical weaknesses

W1. Further Quantitative Evaluation of Persistent Multi-Intent Management

Persistent multi-intent management is one of the central contributions of this paper, and Table 3 provides useful evidence of the Intent Monitor's proactive capabilities. However, evaluating long-running multi-intent interactions is inherently challenging, and more targeted quantitative evaluations would strengthen the experimental evidence. In particular, assessing task success with multiple concurrent proactive intents, intent persistence during interleaved real-time requests, and the correctness of intent creation, updates, cancellations, and priority-based scheduling would be valuable. While evaluating such long-running interactions can be challenging, the current results leave uncertainty about how reliably the framework addresses the persistent multi-intent coordination problem that motivates the work.

W2. Limited Validation of Practical Full-Duplex Interaction

Although the paper provides informative case studies of full-duplex interaction and multi-intent coordination, an accessible demo page or video is not provided.

Questions for authors

Q1. Contribution of the Main Architectural Components

The proposed framework integrates several specialized modules, including the Duplex Controller, Intent Manager, Memory Manager, and unified Intent Monitor. While Appendix A.1 provides an ablation study of the Group-Wise Decision Loss, additional component-level ablations would help clarify the contributions of the individual modules. Could the authors provide component-level ablation studies to clarify the individual contributions of the Duplex Controller, Intent Manager, Memory Manager, and unified Intent Monitor?

Q2. Clarification of the Intent Manager and Intent Monitor Responsibilities

Sections 3.2–3.3 describe the Intent Manager as responsible for intent operations and routing, while the Intent Monitor monitors active proactive intents. However, Appendix B.3 states that ASR transcriptions are first passed to the Intent Monitor, which determines the appropriate intent operation and identifies the corresponding intent. Could the authors clarify the actual execution pipeline and the division of responsibilities between these two modules?

2 comments
Reviewer_BRBcOpenReview ↗

Official Comment by Reviewer BRBc

Thank you for the detailed responses and additional experiments. I appreciate the authors' efforts to address my concerns, particularly through the concurrent proactive monitoring and task-forgetting evaluations.

I could not find a metric explicitly called "Task-Forgetting Rate" in the original OmniInteract paper. I noticed that OmniInteract reports "Missed Outer" rather than "Task-Forgetting Rate." Could the authors clarify whether this metric was defined specifically for the rebuttal? In particular, does it only consider whether a response is generated for each registered task, or does it also account for response correctness and timing?

I understand that constructing a comprehensive benchmark for persistent multi-intent management is challenging. Nevertheless, including the additional quantitative evaluations and the planned demo page in the revised materials would help the broader research community better understand and assess the framework's capabilities in continuous multi-intent interactions.

Reviewer_BRBcOpenReview ↗

Official Comment by Reviewer BRBc

Thank you for the detailed clarification and additional experimental results. I appreciate the authors' efforts to address my concerns, particularly regarding the task-forgetting evaluation.

The explanation of the relationship between "Task-Forgetting Rate" and "Missed Outer" adequately addresses my remaining question. I will maintain my positive assessment of the paper.

Thank you again for your constructive responses.

Reviewer_JbtCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StreamPro-Duplex, a full-duplex multimodal agent framework designed for always-on streaming interaction. The framework incorporates a dedicated duplex controller for managing full-duplex interactions and an Intent Monitor for detecting proactive user intent. The overall system adopts a three-layer architecture comprising seven specialized modules. The authors evaluate the proposed framework across full-duplex interaction and streaming visual interaction tasks.

Critical strengths

The system is carefully designed, with each module serving a clearly defined purpose and motivated by specific challenges in streaming multimodal interaction. The modular architecture provides a structured approach to integrating different capabilities required for always-on interaction.

Critical weaknesses
  1. Incomplete evaluation and missing baselines. The experimental evaluation lacks comparisons with several closely related approaches. Specifically, the duplex controller should be compared against recent semantic VAD methods, such as SoulX-Duplug [1]. For streaming interaction tasks, relevant baselines include Gander [2], Realtime-Venus [3], and Joy-VL-Realtime [4]. Although Gander and Realtime-Venus may be considered concurrent works, the authors have already cited these studies in the related work section and explicitly followed aspects of Gander's design, as described in Appendix B. Therefore, I believe quantitative comparisons with these systems are necessary to establish the effectiveness of the proposed framework.

  2. Potentially unfair comparison and incomplete latency analysis. The comparison with MiniCPM-o appears somewhat unfair, as MiniCPM-o is a single 9B model, whereas StreamPro-Duplex incorporates four 35B models and one 30B model within its framework. This substantial difference in model scale and computational requirements should be explicitly considered when interpreting the results. Furthermore, the authors only report the latency of individual modules, whereas the end-to-end interaction latency of the complete agent should also be evaluated to demonstrate its practical responsiveness in real-time streaming scenarios.

[1] Yan, Ruiqi, et al. "Soulx-duplug: Plug-and-play streaming state prediction module for realtime full-duplex speech conversation." arXiv preprint arXiv:2603.14877 (2026). [2] Ji, Shengpeng, et al. "Omni Interaction Agent Technical Report." arXiv preprint arXiv:2609.08977 (2026). [3] Zhao, Ruixiang, et al. "Realtime-Venus: A full-duplex interaction system with asynchronous delegation." arXiv preprint arXiv:2609.13814 (2026). [4] Yao, Dingyu, et al. "JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence." arXiv preprint arXiv:2606.14777 (2026).

Questions for authors
  1. Could the authors provide additional quantitative comparisons with the relevant baselines mentioned above?

  2. Could the authors report the end-to-end interaction latency of the complete framework, rather than only the latency of individual modules, to better assess its real-time performance?

  3. What are the key architectural and functional differences between StreamPro-Duplex and Gander/Realtime-Venus? In particular, what capabilities does StreamPro-Duplex introduce that are not already supported by these existing full-duplex agent frameworks?

1 comment

Dear Reviewer JbtC,

Thank you again for your valuable feedback. We would like to kindly check whether our previous response has addressed your concerns, or if there are any remaining questions or issues that we could further clarify.

We would greatly appreciate any additional feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.