- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2(−1)
- Soundness
- 2
Read review
Paper summary
This paper provides OmniPersona, a system that can understand user input audio/video and give an appropriate response in both vision and audio. It integrates a frozen MLLM and a fine-tuned joint audio-video generator. The MLLM focuses on reasoning and generating explicit text tokens and implicit learnable tokens to guide the audio-video generator. To avoid drifting, the paper uses diffusion forcing and injects reference signals for the generator, which makes the system applicable to multi-turn interaction.
Critical strengths
-
This paper provides a good integration of an MLLM and an audio-video generation model. It is not simply stitching components together. The authors incorporated several components, allowing OmniPersona to vastly surpass the rudimentary, workflow-style combination of Qwen2.5-Omni and OVI.
-
I noticed in the demo provided by the author that the model is capable of generating interjections like "yeah." This makes the responses feel very authentic. That's really cool.
-
I think the author's various designs to mitigate the drift problem are highly intuitive. Dedicated designs are implemented for both the audio and video modalities, and the effectiveness of each component is demonstrated through ablation studies.
Critical weaknesses
-
I feel the comparative methods included are somewhat limited. I would like to offer two comparison methods: GLARE [1], a paper presented at NeurIPS 2026 that is very similar to OmniPersona; and Ex-Omni-2D [2], a method open-sourced in early August that shares a very similar design to OmniPersona.
-
Although this paper proposes several sound design elements, I believe the performance improvements over OmniResponse [3] are relatively limited. Furthermore, the method used to bridge Qwen is not entirely novel; a similar design was employed in Ex-Omni-2D.
-
I noticed a lot of unnatural jittering in the demo. For instance, faces or shoulders are constantly shaking. What is causing this? The authors should provide a detailed analysis.
Reference:
[1] Liao Z, Suh Y, Ouyang Y, et al. GLARE: Generating Listening Heads with Appropriate Reactions[J]. arXiv preprint arXiv:2609.40317, 2026.
[2 ] Zhang H, Li Z, Tang X, et al. Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence[J]. arXiv preprint arXiv:2608.10720, 2026.
[3] Luo C, Wang J, Li B, et al. Omniresponse: Online multimodal conversational response generation in dyadic interactions[J]. Advances in Neural Information Processing Systems, 2026, 38: 125869-125891.
Questions for authors
All my questions are included in Critical Weaknesses.
At present, I consider this paper to be right on the borderline between acceptance and rejection. However, I cannot assign a score of 2.5. I welcome the authors to provide a rebuttal; I will carefully consider the response and the comments from other reviewers before deciding on the final score.
5 comments
Response to Reviewer 8AZ6 (2/3)
W2a: The method used to bridge Qwen is not entirely novel; a similar design is used in Ex-Omni-2D
We respectfully clarify that the two bridging mechanisms differ fundamentally:
- Ex-Omni-2D connects three modules in sequence. The LLM hidden states are passed only to a separate speech generator (initialized from a TTS model), which produces speech units. The video generator is then conditioned on these generated speech units and on an explicit, discrete text plan (five fields covering scene, emotion, movement style, and motion) encoded by a text encoder. Audio and video therefore cannot influence each other; the influence is one-directional: the video depends on the generated speech, whereas the speech is produced before the video and is not affected by it.
- OmniPersona uses implicit, continuous learnable queries. They read the multimodal dialogue context, including the user's facial and vocal behavior, inside a frozen MLLM and directly condition a joint audio-video generator. Audio and video are generated together in one model and influence each other through bidirectional fusion attention, with the memory conditioning both at once.
We do not claim learnable queries themselves as our contribution; we build on MetaQueries and cite it. Our contribution is to use them as a non-verbal memory of the multi-turn audio-visual dialogue, which is essential for listening behavior, and to combine this memory with masked fusion attention and an anti-drifting design, which together make multi-turn joint audio-video generation stable.
To test whether an explicit text description could replace our learnable queries, we ran a new ablation on our own model: the MLLM reads the dialogue context and describes in text how the avatar should emote and react, and we append this description to the text prompt of OVI, in the spirit of a textual plan. Results on SeamlessInteraction:
| Conditioning of OVI | ↓ | Exp. FD ↓ |
|---|---|---|
| Text + learnable queries (Ours) | 0.31 | 14.28 |
| Text + MLLM description of the avatar's emotion and reaction | 0.47 | 18.13 |
| Text only (w/o MQ) | 0.62 | 19.04 |
The explicit description helps over text only, but learnable queries remain clearly better. The difference is most visible in expression FD, which the description reduces by only 5% and the learnable queries by 25%. Continuous queries capture the intensity and timing of reactions that a discrete description loses. The linear probes in Appendix D.1 are consistent with this: the memory queries encode substantially more decodable affective information than text (emotion Macro-F1 0.554 vs. 0.435; arousal MAE 0.066 vs. 0.123).
W2b: The performance improvement over OmniResponse is relatively limited
We would like to put the numbers in Tab. 3 in context:
- Text metrics mainly reflect the dialogue ability of the MLLM, which is not our main contribution. Open-ended dialogue also has only one reference response, which compresses the METEOR/ROUGE differences for all methods. We still improve METEOR (0.141 → 0.151) and BERTScore (0.806 → 0.820).
- Audio-visual synchronization, which is our focus, improves substantially: LSE-D drops from 9.56 to 5.59 (−42%), even below the ground truth (8.96), whose audio-video misalignment is shown in the supplementary video.
- Audio naturalness is bounded by the ground-truth recordings (UTMOSv2 1.56); we narrow the gap to the ground truth from 0.15 to 0.06 (1.41 → 1.50).
- Video quality improves on ~70 s clips (FVD 314.94 → 284.87).
More importantly, the two methods are qualitatively different. OmniResponse renders low-dimensional, animation-based representations without joint audio-video generation or voice customization, whereas OmniPersona generates photorealistic video with synchronized, voice-cloned speech (Tab. 1, supplementary video). The OmniResponse benchmark does not measure identity preservation, voice similarity, or listening behavior, which is where our advantages are largest (Tab. 4, Tab. 5).
Response to Reviewer 8AZ6 (3/3)
W3: Unnatural jittering in the demo (faces or shoulders constantly shaking)
Thank you for the careful inspection. We explain below where this motion comes from.
The motion reflects natural behavior in the training data. SeamlessInteraction records participants who stand throughout long face-to-face conversations. Without any support, standing people naturally sway, so their heads and shoulders keep moving over the course of a conversation. Our training clips are cropped from the top of the head to just below the shoulders, so this whole-body sway appears in the crop as head and shoulder motion, which OmniPersona learns and reproduces.
Why it can look like shaking. The demo contains both shoulder-level crops and tighter face-only crops. Since only part of the body is visible, the viewer cannot see the body movement that causes the motion. In face-only crops, the same sway also occupies a larger part of the frame and therefore looks more pronounced. Viewed as the upper part of a standing person, however, the motion is consistent and natural.
Evidence. The corresponding ground-truth clips in the supplementary video show similarly large sway, so this motion reflects real behavior rather than a generation artifact. Quantitatively, OmniPersona achieves the lowest pose FD in Tab. 4 (0.04, vs. 0.06 for LiveAvatar and 0.09 for OmniAvatar), i.e., its head rotation and position stay closest to those of the real person. Since this behavior is learned from data, a steadier avatar, for example a seated presenter, could be obtained by training on recordings of seated speakers.
We hope these responses and the new results address your concerns. We would be grateful if you would take them into account in your final assessment, and we are happy to clarify anything further.
Response to Reviewer 8AZ6 (1/3)
We sincerely thank Reviewer R3 for the detailed review, for appreciating our anti-drifting designs and the natural backchannels (e.g., "yeah") in the demo, and for keeping an open mind about the final score. We address each concern below.
W1: Comparison with GLARE and Ex-Omni-2D
Thank you for pointing out these two relevant works. Both are concurrent under the ICLR 2027 Reviewer Guidelines: Ex-Omni-2D is an arXiv-only paper posted within two months before the deadline (v1: Aug 11, 2026), and GLARE appeared after the deadline (v1: Sep 30, 2026). We nevertheless find a careful technical comparison useful, and it shows that the three works address substantially different problems:
GLARE studies when and how a listener reacts to speaker audio. At the task level, this is a sub-problem of ours, corresponding to the listening state of our setting. OmniPersona additionally covers the speaking state, generates speech jointly with video, and conditions on the multimodal dialogue history, none of which GLARE addresses. Its reaction-event annotations are an interesting direction for evaluating listening behavior.
Ex-Omni-2D is closer in spirit, as it also couples an LLM with speech and video generation. However, its three modules run in sequence: the LLM first writes an explicit, hand-structured text plan; a separate speech generator then produces speech units; and the video generator is driven by these speech units together with the text plan. Its dialogue history enters only the LLM, it does not take the user's video as input, and it does not model listening states. For longer outputs, it distills a streaming Student whose one-latent prefix partially mitigates drift (analyzed over 16 chunks of 0.48 s each), and it reports the remaining drift as a limitation. In contrast, OmniPersona generates audio and video together in one diffusion model, conditions on the audio-visual behavior of both parties, covers both speaking and listening, and is evaluated on ~50 s multi-turn conversations.