acceptodds
Withdrawn from ICLR 2027

Beyond Turn-Taking: Multimodal Continuous Full-Duplex Emotional Interaction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Most existing multimodal conversational agents follow a turn-taking paradigm and typically remain silent while the user is speaking. This limits their ability to maintain perception while continuously generating multimodal feedback, such as facial expressions, conditioned on the user's evolving state and interaction context. We formulate this setting as multimodal continuous full-duplex emotional interaction, in which perception and response generation proceed concurrently. To support this task, we introduce **ContinuEI**, a dataset containing 30.24 hours of temporally aligned interactions from 1,072 dialogues and 12,303 sentences. It pairs users' visual-acoustic affective cues with system-side text, speech, and 3D facial motion, preserving the temporal correspondence between evolving user states and multimodal system responses. We further develop **FuSE**, an end-to-end **Fu**ll-duplex **S**treaming **E**motional interaction framework built upon Qwen3-Omni to provide a low-latency baseline for this task. FuSE incorporates an Actor module for continuous facial behavior generation, together with spatio-temporal attention and dual-stream conditional modeling. Experiments show that FuSE improves interaction naturalness and reduces response latency compared with conventional cascaded turn-taking baselines. These results support multimodal continuous emotional interaction as a promising task for studying continuous perception and multimodal response generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Continuous Emotional Interaction, Full-Duplex Streaming Interaction, Speech and 3D Facial Motion Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pe9eOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This article addresses the limitations of traditional conversational agent alternating interaction: AI remains silent when the user speaks, lacking real-time emotional feedback. It proposes a new task of multimodal continuous full duplex emotional interaction, which requires the agent to synchronously complete perception and multimodal feedback output.

Critical strengths

1.The article constructs a time-series aligned dataset called ContinuEI. 2.Based on Qwen3-Omni, an FuSE framework is proposed, which adds an Actor module and utilizes dual stream conditional modeling and spatiotemporal attention to achieve low latency joint generation of text, speech, and facial animations. 3.Experiments have shown that FuSE improves interaction naturalness and reduces response delay compared to cascaded baselines.

Critical weaknesses

1.The characteristics of the dataset proposed in the article are not obvious, and compared with multiple existing head datasets, there are more head data related to emotions. The citation of the article is also incomplete, lacking many related head motion datasets. 2.The method of the article lacks innovation and does not have a special design based on the characteristics of the data.

Questions for authors

1.The characteristics of the dataset proposed in the article are not obvious, and compared with multiple existing head datasets, there are more head data related to emotions. The citation of the article is also incomplete, lacking many related head motion datasets. 2.The method of the article lacks innovation and does not have a special design based on the characteristics of the data. 3.As a head motion related job, there is almost no display of dynamic effects. This kind of work cannot be accepted. 4.The experimental comparison method is too limited, and many well-known open-source methods have not added comparisons.

Reviewer_1WFeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines multimodal continuous full-duplex emotional interaction, where full-duplex means that perception and expression run concurrently, not that the system handles barge-in. The agent should watch and listen while continuously generating text, speech, and 3D facial motion. ContinuEI is built from MESC: 30.24 hours, 1,072 dialogues, and 12,303 sentences. The proposed framework achieves Contextual coherence is 3.03 and emotional consistency is 2.41, above Gemini 3.5 Flash and Qwen3-Omni. Empathy is 2.03, slightly below Gemini’s 2.05. Total latency to the first face frame is 906 ms, close to Qwen+A2F at 908 ms and far below non-streaming cascades.

Critical strengths
  1. The task is framed clearly. The listener’s face during the user’s speech is treated as a timed generation problem. ContinuEI puts user audio and video, system speech, and blendshapes on one timeline, and it masks the therapist’s face so the model cannot see the answer.

  2. This paper provided a dataset with multimodal data.

Critical weaknesses
  1. The paper does not compare with closely related listener and dyadic face work. OmniResponse [1], INFP [2], and DualTalk [3] all generate listener or dyadic faces from the speaker’s stream, and earlier listener-head work [4, 5, 6, 7] also conditions on the speaker. FuSE cites none of these methods. Without a comparison on the same setting, I cannot tell what FuSE adds beyond the counseling data, the ARKit output, and the listener-to-responder switch.

  2. There is no metric for response consistency. Text is scored by GPT alone. Speech metrics check each modality on its own (WER, speaker similarity, EmoACC). Face metrics compare against the synthetic target, and the responder is excluded because its words differ. Naturalness was rated one modality at a time, with evaluators hearing only the speech or seeing only the face. CEC checks agreement between the agent’s own speech and face, not with the user. None of the metrics asks whether the text, voice, and face together form one appropriate response to the user’s current face and speech.

  3. There is no demo or visualization of the interaction. The paper has no demo video, no project page, and no timing plot of the listener and responder streams. Figure 3 is a set of still frames, so a reader cannot check timing, lip sync, or whether the face responds to the user at the right moment.

  4. The experiments are small and mostly in-distribution. The end-to-end claims rest on 0.62 hours and 29 dialogues, with no intervals. Most results are module-level scores. The one cross-dataset check (BEAT) covers only the face module.

[1] Cheng Luo, Jianghui Wang, Bing Li, Siyang Song, and Bernard Ghanem. OmniResponse: Online multimodal conversational response generation in dyadic interactions. NeurIPS, 2025. arXiv:2505.21724.

[2] Yongming Zhu, Longhao Zhang, Zhengkun Rong, Tianshu Hu, Shuang Liang, and Zhipeng Ge. INFP: Audio-driven interactive head generation in dyadic conversations. CVPR, 2025. arXiv:2412.04037.

[3] Ziqiao Peng, Yanbo Fan, Haoyu Wu, Xuan Wang, Hongyan Liu, Jun He, and Zhaoxin Fan. DualTalk: Dual-speaker interaction for 3D talking head conversations. CVPR, 2025. arXiv:2505.18096.

[4] Evonne Ng, Hanbyul Joo, Liwen Hu, Hao Li, Trevor Darrell, Angjoo Kanazawa, and Shiry Ginosar. Learning to listen: Modeling non-deterministic dyadic facial motion. CVPR, 2022.

[5] Mohan Zhou, Yalong Bai, Wei Zhang, Ting Yao, Tiejun Zhao, and Tao Mei. Responsive listening head generation: A benchmark dataset and baseline. ECCV, 2022. arXiv:2112.13548.

[6] Cheng Luo, Siyang Song, Weicheng Xie, Micol Spitale, Zongyuan Ge, Linlin Shen, and Hatice Gunes. ReactFace: Online multiple appropriate facial reaction generation in dyadic interactions. IEEE TVCG, 2024. arXiv:2305.15748.

[7] Siyang Song, Micol Spitale, Cheng Luo, Germán Barquero, Cristina Palmero, Sergio Escalera, Michel Valstar, Tobias Baur, Fabien Ringeval, Elisabeth André, and Hatice Gunes. REACT2023: The first multiple appropriate facial reaction generation challenge. ACM MM, 2023. arXiv:2306.06583.

Questions for authors
  1. Please compare FuSE with OmniResponse, INFP, and DualTalk on the same listener and role-switch setting. If those systems already cover this behavior, narrow the new-task claim to the counseling data and the ARKit output.

  2. Please add an evaluation of response consistency. Given the user’s current face and speech, do the agent’s text, voice, and face agree with each other and fit the user? Describe the protocol, the raters, inter-rater agreement, and whether raters saw all three modalities together.

  3. Please provide a demo video of full exchanges, with the user video, the agent’s speech, and the rendered blendshapes on one timeline, plus a timing plot of the listener and responder streams.

  4. Do the synthetic blendshapes agree with the MESC emotion labels, or only look smooth? What does the Actor achieve when Audio2Face is removed and only visual reconstruction is used?

A side-by-side definition against Moshi and FLEXI would help readers, but it is a suggestion, not a condition of acceptance.

Reviewer_4BPeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper claims that existing multimodal emotional dialogue systems follow a turn-taking paradigm and remain silent while the user is speaking, and this kind of interaction gap disrupts the continuity of emotional feedback and limits the ability of agents to maintain sustained emotional engagement. Therefore, the authors aimed to develop a full-duplex emotional interaction generation approach. First, they built a dataset for this goal, called ContinuEI, with the help of a multi-step workflow, using insightface, mediapipe, audio2face, etc. After that, they propose an end-to-end full-duplex streaming emotional interaction framework, FuSE, which extends Qwen3-Omni with an Actor module to enable continuous full-duplex emotional interaction. Experiments conducted on ContinuEI shows that the proposed FuSE achieves competitive interaction naturalness and first-packet latency relative to Gemini3.5Flash and Qwen3Omni.

Critical strengths
  1. This paper not only builds a dataset for the continuous multimodal emotional interaction, but also proposes a framework to enable agents to support continuous and emotionally coherent interaction. It makes contributions from both data and method perspectives.
  2. This paper is well-written and has a good presentation.
  3. Motivated by the full-duplex speech dialogue research, this paper aims to develop a full-duplex multimodal emotional interaction method. The motivation is clear and makes sense.
Critical weaknesses
  1. The main challenges this paper aims to address are unclear. The interaction gap is a real limitation and it hurts the continuity. However, I was not clear about what challenges I might face while attempting to address this limitation.
  2. Similar with weakness 1, the challenges the ContinuEI dataset aims to address are not clear. Is there any existing dataset match the requirements of the full-duplex emotional interaction? The related work section misses dataset survey, which exacerbated the problem.
  3. The experiments in this paper are based on a synthetic dataset, ContinuEI, only. Can the conclusions drawn in this paper be generalized to other datasets? The missing experiments on other datasets, especially on some real data, not synthetic data, hurt the contribution of this paper.
  4. The FuSE is based on Qwen3-Omni and extends it with an Actor module. But this paper miss an ablation study for this critical module.
Questions for authors
  1. Please clarify the challenges this paper aims to address, from the dataset perspective and method perspective.
  2. How generalizable is this method? Please discuss the generalization of the proposed method on some other datasets.
  3. How well does the method perform without the Actor module?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.