acceptodds
Under review as a conference paper at ICLR 2027

OmniPersona: Persona-Embodied Agents for Multimodal Conversation Interaction

Abstract

Current interactive talking avatar models remain fundamentally unaware of dialogue context, requiring explicit textual or audio driving signals for each response, thereby functioning more like video renderers than autonomous avatar agents. Inspired by recent advances in unified understanding and generation frameworks, we present OmniPersona, an agentic model that integrates a pretrained multimodal large language model (MLLM) with an adapted joint audio-video generator. Our framework unifies dialogue understanding with synchronized audio-video avatar generation within a single architecture. Central to this design are learnable queries that compress historical multimodal context into a fixed-size non-verbal memory. Together with verbal content, this memory enables our generator to produce more responsive avatar behaviors, particularly in the listening state, which remains underexplored in prior work centered on talking scenarios. Additionally, we incorporate a suite of anti-drifting techniques, including a masked fusion attention design, to support persona consistency (audio-visual identity and synchronization) across multi-turn conversations. This attention design enables customized voice conditioning without compromising audio-video consistency. Extensive experiments on three datasets and human evaluation demonstrate that OmniPersona achieves superior performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human interaction in AI systems, Human conversational generation, Interactive avatars

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_8AZ6OpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

This paper provides OmniPersona, a system that can understand user input audio/video and give an appropriate response in both vision and audio. It integrates a frozen MLLM and a fine-tuned joint audio-video generator. The MLLM focuses on reasoning and generating explicit text tokens and implicit learnable tokens to guide the audio-video generator. To avoid drifting, the paper uses diffusion forcing and injects reference signals for the generator, which makes the system applicable to multi-turn interaction.

Critical strengths
  1. This paper provides a good integration of an MLLM and an audio-video generation model. It is not simply stitching components together. The authors incorporated several components, allowing OmniPersona to vastly surpass the rudimentary, workflow-style combination of Qwen2.5-Omni and OVI.

  2. I noticed in the demo provided by the author that the model is capable of generating interjections like "yeah." This makes the responses feel very authentic. That's really cool.

  3. I think the author's various designs to mitigate the drift problem are highly intuitive. Dedicated designs are implemented for both the audio and video modalities, and the effectiveness of each component is demonstrated through ablation studies.

Critical weaknesses
  1. I feel the comparative methods included are somewhat limited. I would like to offer two comparison methods: GLARE [1], a paper presented at NeurIPS 2026 that is very similar to OmniPersona; and Ex-Omni-2D [2], a method open-sourced in early August that shares a very similar design to OmniPersona.

  2. Although this paper proposes several sound design elements, I believe the performance improvements over OmniResponse [3] are relatively limited. Furthermore, the method used to bridge Qwen is not entirely novel; a similar design was employed in Ex-Omni-2D.

  3. I noticed a lot of unnatural jittering in the demo. For instance, faces or shoulders are constantly shaking. What is causing this? The authors should provide a detailed analysis.

Reference:

[1] Liao Z, Suh Y, Ouyang Y, et al. GLARE: Generating Listening Heads with Appropriate Reactions[J]. arXiv preprint arXiv:2609.40317, 2026.

[2 ] Zhang H, Li Z, Tang X, et al. Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence[J]. arXiv preprint arXiv:2608.10720, 2026.

[3] Luo C, Wang J, Li B, et al. Omniresponse: Online multimodal conversational response generation in dyadic interactions[J]. Advances in Neural Information Processing Systems, 2026, 38: 125869-125891.

Questions for authors

All my questions are included in Critical Weaknesses.

At present, I consider this paper to be right on the borderline between acceptance and rejection. However, I cannot assign a score of 2.5. I welcome the authors to provide a rebuttal; I will carefully consider the response and the comments from other reviewers before deciding on the final score.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 8AZ6 (1/3)

We sincerely thank Reviewer R3 for the detailed review, for appreciating our anti-drifting designs and the natural backchannels (e.g., "yeah") in the demo, and for keeping an open mind about the final score. We address each concern below.

W1: Comparison with GLARE and Ex-Omni-2D

Thank you for pointing out these two relevant works. Both are concurrent under the ICLR 2027 Reviewer Guidelines: Ex-Omni-2D is an arXiv-only paper posted within two months before the deadline (v1: Aug 11, 2026), and GLARE appeared after the deadline (v1: Sep 30, 2026). We nevertheless find a careful technical comparison useful, and it shows that the three works address substantially different problems:

GLAREEx-Omni-2DOmniPersona (Ours)
TaskListening head onlySpeaking avatarSpeaking and listening, multi-turn
Role of the (M)LLMFrozen Qwen2-Audio extracts a scalar prosody intensityQwen3-8B (fine-tuned) writes an explicit text "Visual Thought Plan" and passes its response hidden states to the speech generatorFrozen Qwen2.5-Omni; learnable queries compress the multimodal dialogue context into a non-verbal memory
User inputSpeaker audioText and/or speech; no user videoAudio + video of both parties across turns
Speech generationNone (audio is an input)A separate speech generator (initialized from a TTS model) produces speech unitsAudio and video generated jointly in one diffusion model, with a reference voice
Multi-turn audio-visual contextNoDialogue history enters the LLM only; no audio-visual historyYes
Long-horizon driftNot addressedTeacher trained on 5 s clips; a distilled streaming Student uses a one-latent prefix that partially mitigates drift; remaining drift is a stated limitationDedicated anti-drifting suite; ~50 s multi-turn evaluation

GLARE studies when and how a listener reacts to speaker audio. At the task level, this is a sub-problem of ours, corresponding to the listening state of our setting. OmniPersona additionally covers the speaking state, generates speech jointly with video, and conditions on the multimodal dialogue history, none of which GLARE addresses. Its reaction-event annotations are an interesting direction for evaluating listening behavior.

Ex-Omni-2D is closer in spirit, as it also couples an LLM with speech and video generation. However, its three modules run in sequence: the LLM first writes an explicit, hand-structured text plan; a separate speech generator then produces speech units; and the video generator is driven by these speech units together with the text plan. Its dialogue history enters only the LLM, it does not take the user's video as input, and it does not model listening states. For longer outputs, it distills a streaming Student whose one-latent prefix partially mitigates drift (analyzed over 16 chunks of 0.48 s each), and it reports the remaining drift as a limitation. In contrast, OmniPersona generates audio and video together in one diffusion model, conditions on the audio-visual behavior of both parties, covers both speaking and listening, and is evaluated on ~50 s multi-turn conversations.

AuthorsOpenReview ↗

Response to Reviewer 8AZ6 (2/3)

W2a: The method used to bridge Qwen is not entirely novel; a similar design is used in Ex-Omni-2D

We respectfully clarify that the two bridging mechanisms differ fundamentally:

  • Ex-Omni-2D connects three modules in sequence. The LLM hidden states are passed only to a separate speech generator (initialized from a TTS model), which produces speech units. The video generator is then conditioned on these generated speech units and on an explicit, discrete text plan (five fields covering scene, emotion, movement style, and motion) encoded by a text encoder. Audio and video therefore cannot influence each other; the influence is one-directional: the video depends on the generated speech, whereas the speech is produced before the video and is not affected by it.
  • OmniPersona uses implicit, continuous learnable queries. They read the multimodal dialogue context, including the user's facial and vocal behavior, inside a frozen MLLM and directly condition a joint audio-video generator. Audio and video are generated together in one model and influence each other through bidirectional fusion attention, with the memory conditioning both at once.

We do not claim learnable queries themselves as our contribution; we build on MetaQueries and cite it. Our contribution is to use them as a non-verbal memory of the multi-turn audio-visual dialogue, which is essential for listening behavior, and to combine this memory with masked fusion attention and an anti-drifting design, which together make multi-turn joint audio-video generation stable.

To test whether an explicit text description could replace our learnable queries, we ran a new ablation on our own model: the MLLM reads the dialogue context and describes in text how the avatar should emote and react, and we append this description to the text prompt of OVI, in the spirit of a textual plan. Results on SeamlessInteraction:

Conditioning of OVIFIDemo\mathrm{FID_{emo}} ↓Exp. FD ↓
Text + learnable queries (Ours)0.3114.28
Text + MLLM description of the avatar's emotion and reaction0.4718.13
Text only (w/o MQ)0.6219.04

The explicit description helps over text only, but learnable queries remain clearly better. The difference is most visible in expression FD, which the description reduces by only 5% and the learnable queries by 25%. Continuous queries capture the intensity and timing of reactions that a discrete description loses. The linear probes in Appendix D.1 are consistent with this: the memory queries encode substantially more decodable affective information than text (emotion Macro-F1 0.554 vs. 0.435; arousal MAE 0.066 vs. 0.123).

W2b: The performance improvement over OmniResponse is relatively limited

We would like to put the numbers in Tab. 3 in context:

  • Text metrics mainly reflect the dialogue ability of the MLLM, which is not our main contribution. Open-ended dialogue also has only one reference response, which compresses the METEOR/ROUGE differences for all methods. We still improve METEOR (0.141 → 0.151) and BERTScore (0.806 → 0.820).
  • Audio-visual synchronization, which is our focus, improves substantially: LSE-D drops from 9.56 to 5.59 (−42%), even below the ground truth (8.96), whose audio-video misalignment is shown in the supplementary video.
  • Audio naturalness is bounded by the ground-truth recordings (UTMOSv2 1.56); we narrow the gap to the ground truth from 0.15 to 0.06 (1.41 → 1.50).
  • Video quality improves on ~70 s clips (FVD 314.94 → 284.87).

More importantly, the two methods are qualitatively different. OmniResponse renders low-dimensional, animation-based representations without joint audio-video generation or voice customization, whereas OmniPersona generates photorealistic video with synchronized, voice-cloned speech (Tab. 1, supplementary video). The OmniResponse benchmark does not measure identity preservation, voice similarity, or listening behavior, which is where our advantages are largest (Tab. 4, Tab. 5).

AuthorsOpenReview ↗

Response to Reviewer 8AZ6 (3/3)

W3: Unnatural jittering in the demo (faces or shoulders constantly shaking)

Thank you for the careful inspection. We explain below where this motion comes from.

The motion reflects natural behavior in the training data. SeamlessInteraction records participants who stand throughout long face-to-face conversations. Without any support, standing people naturally sway, so their heads and shoulders keep moving over the course of a conversation. Our training clips are cropped from the top of the head to just below the shoulders, so this whole-body sway appears in the crop as head and shoulder motion, which OmniPersona learns and reproduces.

Why it can look like shaking. The demo contains both shoulder-level crops and tighter face-only crops. Since only part of the body is visible, the viewer cannot see the body movement that causes the motion. In face-only crops, the same sway also occupies a larger part of the frame and therefore looks more pronounced. Viewed as the upper part of a standing person, however, the motion is consistent and natural.

Evidence. The corresponding ground-truth clips in the supplementary video show similarly large sway, so this motion reflects real behavior rather than a generation artifact. Quantitatively, OmniPersona achieves the lowest pose FD in Tab. 4 (0.04, vs. 0.06 for LiveAvatar and 0.09 for OmniAvatar), i.e., its head rotation and position stay closest to those of the real person. Since this behavior is learned from data, a steadier avatar, for example a seated presenter, could be obtained by training on recordings of seated speakers.

We hope these responses and the new results address your concerns. We would be grateful if you would take them into account in your final assessment, and we are happy to clarify anything further.

Reviewer_WHNkOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

OmniPersona is a unified framework that combines dialogue understanding with synchronized audio-visual (A-V) avatar generation in a single architecture. Unlike previous systems that treat multi-turn conversation as isolated, audio-driven lip-sync video rendering tasks, OmniPersona supports context-aware interaction, realistic listening behaviors, and multi-turn audio-visual synthesis.

To bridge multimodal LLMs with A-V diffusion models, built on an OVI backbone, OmniPersona compresses historical context into fixed-size learnable queries (Q^\hat{Q}), aligning them with T5 text-speech transcription embeddings via a K-layer Transformer and zero-initialized projection layer. It uses asymmetric fusion cross-attention to prevent acoustic content leakage while enabling voice customization. To solve the long-term generation drift and exposure bias common in autoregressive diffusion models, it introduces a suite of complementary anti-drifting mechanisms: Diffusion Forcing, and Strided RoPE for reference anchoring.

Trained on the SeamlessInteraction dataset, OmniPersona demonstrates superior performance across the SeamlessInteraction, SpeakerVid-5M, and OmniResponse benchmarks, as well as in human evaluations. It significantly outperforms existing baselines in audio-video long-term consistency, multi-turn dialogue dynamics, and responsive listening behavior.

Critical strengths
  • Well-Motivated Architectural Design: The paper introduces OmniPersona, a framework capable of context-aware, audio-synchronized, and multi-turn interactions. Bridging Multimodal LLMs (MLLMs) with Audio-Visual (A-V) diffusion models via learnable historical query compression is technically sound and effective.

  • Extensive and Rigorous Experimental Validation: The proposed methodology is thoroughly supported by comprehensive experiments. The authors demonstrate superior performance across established benchmarks (SpeakerVid-5M and OmniResponse). Crucially, they go beyond standard benchmarks by designing additional experiments to capture long-term conversational dynamics. Furthermore, well-designed ablation studies and positive user study results convincingly justify the individual components and overall efficacy of the approach.

Critical weaknesses

Lack of Controlled Baseline Comparison on the SpeakerVid-5M benchmark: It is unclear whether the cascade baseline (Qwen2.5-Omni + OVI) was fine-tuned on the exact same training dataset as the proposed method. Without strictly controlling for the training data, it is difficult to determine whether the reported performance gains stem from the proposed methodology or simply from differences in the training data.

Questions for authors
  • (Major) Training Data Alignment: Was the cascade baseline (Qwen2.5-Omni + OVI) fine-tuned on the exact same training dataset as the proposed method? If not, how do you isolate algorithmic gains from data differences?

  • (minor) Comprehensiveness of Multi-Turn Dynamics Evaluation: While the paper evaluates long-term video quality, visual-auditory identity consistency, emotional distribution, and facial expressions, are these metrics sufficient to fully capture the dynamics of multi-turn interactions? Could the authors discuss whether other interaction-level aspects, such as turn-taking responsiveness or cross-turn semantic consistency, were considered or should be evaluated?

3 comments
AuthorsOpenReview ↗

Response to Reviewer WHNk (1/2)

We thank Reviewer R2 for the strong support and for highlighting the importance of controlled comparisons.

Major: Was the cascade baseline (Qwen2.5-Omni + OVI) fine-tuned on the same training data? How are algorithmic gains isolated from data differences?

In Tab. 2, Qwen2.5-Omni + OVI uses the publicly released pretrained OVI without fine-tuning, as an off-the-shelf cascade reference. To separate algorithmic gains from data effects, we now report a data-controlled cascade: our w/o MQ variant. It is exactly a Qwen2.5-Omni → text → OVI cascade, but fine-tuned on the same SeamlessInteraction training data, with the same backbone, LoRA configuration, training schedule, and anti-drifting design as OmniPersona. The only difference is that it has no non-verbal memory.

Method (SpeakerVid-5M)Training dataFVD ↓ArcFace ↑Syncconf\mathrm{Sync_{conf}} ↑FIDemo\mathrm{FID_{emo}} ↓SIM-o ↑
SpeakerVid-5MSpeakerVid-5M (in-domain)28.820.7722.6983.220.65
Qwen2.5-Omni + OVIOVI pretraining only18.500.5333.2030.690.42
Ours w/o MQ (data-controlled cascade, new)SeamlessInteraction8.670.7363.4810.520.63
OursSeamlessInteraction5.180.7743.9350.200.65
  • Pretrained OVI → w/o MQ. Fine-tuning brings large gains on all metrics, for two main reasons. First, OVI is pretrained at 960×960, whereas the benchmark is evaluated at 448×448. Applied directly at this lower resolution, the pretrained model produces degraded image quality; fine-tuning adapts it to the test resolution, which improves video quality and audio-video synchronization (FVD 18.50 → 8.67; Syncconf\mathrm{Sync_{conf}} 3.203 → 3.481). Second, the pretrained OVI takes no reference audio and is conditioned only on the first frame, without strided-RoPE reference anchoring. Its voice therefore does not follow the target speaker, and its appearance gradually deviates from the reference identity. With reference audio and an anchored reference image, w/o MQ improves voice similarity and identity preservation (SIM-o 0.42 → 0.63; ArcFace 0.533 → 0.736).
  • w/o MQ → Ours. With identical data, backbone, and training, adding the non-verbal memory improves all five metrics (FVD 8.67 → 5.18; ArcFace 0.736 → 0.774; Syncconf\mathrm{Sync_{conf}} 3.481 → 3.935; FIDemo\mathrm{FID_{emo}} 0.52 → 0.20; SIM-o 0.63 → 0.65). The largest relative gain is in expression quality (FIDemo\mathrm{FID_{emo}} −62%), which matches the role of the memory. This gap is therefore attributable to the method rather than to the data. The same conclusion holds on SeamlessInteraction (Tab. 4: FIDemo\mathrm{FID_{emo}} 0.62 → 0.31).

Two further points indicate that the gains are not due to training data:

  1. OmniPersona is trained only on SeamlessInteraction and evaluated on SpeakerVid-5M zero-shot, whereas the SpeakerVid-5M baseline is trained in-domain on SpeakerVid-5M. Its numbers are those reported by its authors, obtained in their own evaluation at the native resolution of their model, rather than re-run under our setting. If anything, this comparison favors that baseline.
  2. In the same-response-text control (Appendix D.2), all methods use identical response text, and OmniPersona remains the best on all metrics.
AuthorsOpenReview ↗

Response to Reviewer WHNk (2/2)

Minor: Are the current metrics sufficient to capture multi-turn dynamics? What about turn-taking responsiveness and cross-turn semantic consistency?

We agree that multi-turn interaction involves more than per-frame quality. Below we address turn-taking, cross-turn consistency, and context dependence with two new measurements and existing results, and then note the remaining gap.

Turn-taking responsiveness (new). We divide each SeamlessInteraction test conversation into 1 s windows and label each window as speaking or listening. A window is labeled as speaking when voice activity covers more than half of it, as detected by the same VAD model on the generated and the ground-truth agent audio. Since speaking and listening windows are imbalanced, we compare the generated and ground-truth labels with balanced accuracy (the mean of the per-class accuracies; chance level 50%). OmniPersona reaches 80.6%, indicating that, in most cases, the agent starts and stops speaking in line with the conversation at a 1 s resolution. Beyond this binary state, the agent also produces short backchannels such as "yeah" while listening (supplementary video), which this metric does not capture.

Cross-turn semantic consistency (existing results). The textual response is generated from the full dialogue history (Appendix B.3), so each turn is conditioned on all previous turns. The OmniResponse benchmark (Tab. 3) evaluates exactly this multi-turn setting on ~70 s conversations: its text metrics compare each generated response with the ground-truth reply given the preceding dialogue, and OmniPersona outperforms OmniResponse on METEOR (0.151 vs. 0.141) and BERTScore (0.820 vs. 0.806). On the non-verbal side, the affective information in the memory remains stable across turns (emotion-probe Macro-F1 0.651 at Turn 1 vs. 0.642 at Turn 10; Appendix D.1).

Context dependence (new). To test whether the generated reactions follow the dialogue context, we ran a counterfactual context-swap test: we keep the reference image and voice, the prefix, and the textual response fixed, and replace only the dialogue context fed into the non-verbal memory. All metrics are computed within listening intervals against the ground-truth listener of the original sample:

Context fed to the memoryFIDemo\mathrm{FID_{emo}} ↓Expression FD ↓
Matched context (Ours)0.3114.28
Context from a random other conversation0.6319.25
Emotion-inverted context1.5422.78
For reference: no memory (w/o MQ, Tab. 4)0.6219.04

A context from a random other conversation is about as poor as having no memory at all, and an emotion-inverted context is far worse. Since all other inputs are identical, the generated reactions are driven by the dialogue context.

Remaining gap: full-duplex behavior. The test conversations are natural dyadic interactions, but current metrics, including ours, do not explicitly evaluate full-duplex behaviors such as interrupting, being interrupted, and overlapping speech. Designing such metrics is an important direction for future work.

We thank the reviewer again for the constructive suggestions, which have strengthened our evaluation.

Thank you to the authors for the thorough and constructive rebuttal. The newly introduced data-controlled cascade baseline (Ours w/o MQ), fine-tuned on the exact same SeamlessInteraction training data, convincingly isolates the algorithmic gains of the non-verbal memory module from training data differences on SpeakerVid-5M. In addition, the new evaluations on turn-taking responsiveness and counterfactual context dependence—together with the candid discussion of remaining limitations such as full-duplex behaviors—satisfactorily address my questions regarding multi-turn interaction dynamics. All of my concerns have been well resolved, and I am happy to maintain my positive rating for clear acceptance.

Reviewer_ZgHBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a unified framework named OmniPersona for interactive avatar agents generation. It integrates a pretrained multimodal large language model (MLLM) with a joint audio-video diffusion generator for context-aware, multi-turn conversational interaction. The framework uses learnable queries to compress historical multimodal context into a fixed-size non-verbal memory, then this compressed memory is combined with the generated textual response to condition synchronized audio-video generation. To improve identity consistency across multiple turns, the authors introduce several anti-drifting techniques, including diffusion forcing, masked fusion cross-attention, and strided RoPE. The method is evaluated on three datasets, including OmniResponse, SpeakerVid-5M, and SeamlessInteraction. Experimental results and human evaluations show improvements in video quality, audio-video synchronization, identity preservation, and listening behavior.

Critical strengths
  1. This paper unifies dialogue understanding with synchronized audio video avatar generation within a single architecture.

  2. The proposed learnable queries compress historical audio visual context into a compact memory condition. Combining this non-verbal memory with the textual response enhances the naturalness and coherence of generated content.

  3. The experimental results show improvements in audio-video synchronization, facial expression quality, and identity consistency over existing baselines.

Critical weaknesses
  1. The paper claims to model historical multimodal context for multi-turn conversational interaction. However, the non-verbal memory only encodes 10 seconds of audio-video context from the previous conversational turn (Appendix B.3). I have concerns that if some important information is outside of the 10 seconds context length, the generated response maybe not good. Could the authors give more discussion about it?

  2. Most experiments in the paper focus on video quality, identity consistency, and audio-video synchronization. There is limited evaluation of whether the generated facial expressions and listening behaviors correctly reflect the dialogue context. Because the paper claims to enable context-aware conversational interaction, so it's important to prove it in the experiments.

Questions for authors
  1. If certain important information falls outside the 10 second context window, will the quality of the generated response be affected?

  2. Is there any way to automatically evaluate whether the generated facial expressions and listening behaviors correctly reflect the dialogue context?

  3. I'm curious that how much improvement comes from the proposed compression mechanism rather than simply providing additional audio-visual context? Could the authors compare learnable queries with simpler context-conditioning methods? Or give some discussions about it.

3 comments
AuthorsOpenReview ↗

Response to Reviewer ZgHB (1/2)

We thank Reviewer R1 for the positive assessment and the insightful questions. We address each point below.

W1 / Q1: The non-verbal memory encodes only 10 s of audio-visual context. What if important information lies outside this window?

The 10 s window constrains only how the avatar behaves, not what it says, and we find that it loses little in practice. We first explain why, and then test the reviewer's case directly.

The generator receives two conditions: the textual response determines what the avatar says, and the non-verbal memory determines how it speaks and listens. The memory encodes the most recent audio-visual context of both parties (10 s by default; trained with variable 5–20 s windows).

  • Speaking. Qwen2.5-Omni generates the textual response TST^S from the full dialogue history (Appendix B.3), so long-range semantic information, such as topics or facts from several turns earlier, is preserved in what is said. The memory affects only how it is said, i.e., the prosody and facial expression.
  • Listening. The text condition is simply the fixed word "listening", which carries no information about the context. How the avatar listens, i.e., its facial affect and backchannel timing, is determined entirely by the memory.

The reviewer's concern therefore applies to how the avatar speaks and listens, and most directly to listening, where the memory is the only condition that changes with the context. Even there, information outside the 10 s window is not entirely lost, for two reasons:

  1. Earlier context is carried forward implicitly. The context encoded by the memory includes the agent's own previously generated audio and video. Since this behavior was itself generated from the earlier context, information propagates across turns through the autoregressive chain.
  2. Affect is continuous across segments. Each new segment starts from prefix frames taken from the end of the previously generated audio and video.

The paper already supports this on the full test set: extending the memory window from 10 s to 20 s brings no further gain (Fig. 4 / Tab. 12; FID$_{emo}$ 0.31 → 0.32), which suggests that the affective cues relevant to the next response are mostly available in the recent context.

New experiment: memory window on long-range cases. To test the reviewer's case directly, we build a long-range subset of SeamlessInteraction. We select the 20 of the 60 test clips that contain segments whose salient affective event, i.e., the user's arousal peak taken from the official Imitator annotations of SeamlessInteraction, lies more than 10 s before the start of the segment. We generate each full ~50 s conversation exactly as in the main evaluation and compute FID$_{emo}$ only on these segments, for which the event is outside the default window when they are generated. We then vary the memory window at inference:

Memory windowFID$_{emo}$ ↓ (all 60 clips)FID$_{emo}$ ↓ (long-range segments from 20 clips)
10 s (default)0.310.35
20 s0.320.33
40 s0.390.43

We draw two observations:

  1. The 10 s and 20 s windows perform comparably, also on long-range cases. They differ by 0.01 on the full set and by 0.02 on the long-range segments. Even when the salient event lies outside the default 10 s window, the default setting loses little, which is consistent with the two properties above. We keep 10 s as the default because a longer window brings no clear benefit but increases the MLLM encoding cost.
  2. Enlarging the window further, to 40 s, does not help. Both sets degrade (0.39 / 0.43), likely because 40 s lies outside the 5–20 s range seen during training and because, with a fixed set of 128 queries, older content may dilute the recent cues.

We agree that very long-range non-verbal dependencies, such as a mood established several minutes earlier, remain a limitation. A scalable remedy, which we leave for future work, is a recurrent memory that carries the previous turn's memory tokens Mn−1M^{n-1} forward at constant cost. We have added this discussion to the limitations.

AuthorsOpenReview ↗

Response to Reviewer ZgHB (2/2)

W2 / Q2: Is there an automatic way to evaluate whether the generated expressions and listening behaviors correctly reflect the dialogue context?

Yes. Our existing metrics already use the context as a reference, and we add a counterfactual test as direct causal evidence.

Existing metrics. In Tab. 4, FID$_{emo}$ and expression/pose FD are computed within listening intervals against the ground-truth listener's reaction to the same context. They therefore measure how close the generated reaction is to the real human reaction for that specific context. The EA score of the human study (Tab. 5) also evaluates contextual appropriateness directly.

New: counterfactual context-swap test. We keep the reference image and voice, the prefix, and the textual response fixed, and replace only the dialogue context fed into the non-verbal memory:

Context fed to the memoryFID$_{emo}$ ↓Expression FD ↓
Matched context (Ours)0.3114.28
Context from a random other conversation0.6319.25
Emotion-inverted context1.5422.78
For reference: no memory (w/o MQ, Tab. 4)0.6219.04

All metrics are computed within listening intervals against the ground-truth listener of the original sample.

A context from a random other conversation **roughly doubles FID${emo}∗∗(0.31→0.63)andincreasesexpressionFDby35** (0.31 → 0.63) and increases expression FD by 35% (14.28 → 19.25), which is about as poor as removing the memory altogether (*w/o MQ*: 0.62 / 19.04). An emotion-inverted context is far worse than having no memory at all: FID{emo}$ rises about fivefold (0.31 → 1.54) and expression FD by 60% (14.28 → 22.78). Since all other inputs are identical, these differences can only come from the dialogue context. The model does not simply ignore an unsuitable context: its reactions are driven by the dialogue context, not only by the reference image or the prefix. Together with the linear-probe analysis (Appendix D.1), these results provide an automatic evaluation of context appropriateness.

Q3: How much of the gain comes from compression rather than from simply providing more audio-visual context? Comparison with simpler conditioning methods?

We compare the learnable queries with a simpler way of providing the same audio-visual context on SeamlessInteraction:

Conditioning of OVI (10 s context)FID$_{emo}$ ↓Exp. FD ↓
Text + 128 learnable queries (Ours)0.3114.28
Text + MLLM description of the avatar's emotion and reaction (new)0.4718.13
Text only (w/o MQ)0.6219.04
  • Explicit description (simpler conditioning). The MLLM reads the same dialogue context and describes in text how the avatar should emote and react (e.g., "smiles and nods slightly while listening"); we append this description to the text prompt of OVI. This passes the context to the generator through the simplest possible channel, without any learned compression. It improves over text only (FID$_{emo}$ 0.62 → 0.47) but remains clearly worse than our learnable queries (0.31). The gap is larger in expression FD, which the description reduces by only 5% (19.04 → 18.13) and the learnable queries by 25% (→ 14.28). This suggests that a discrete description conveys the coarse emotion category, but not fine-grained details such as the intensity and timing of a reaction.
  • Degree of compression. Our capacity sweep (Fig. 4) varies how strongly the context is compressed. With a 10 s context, FID$_{emo}$ improves from 0.74 (32 queries) to 0.49 (64) and 0.31 (128), while 192 queries bring no further gain (0.34). 128 tokens are therefore enough to retain the relevant affective information.

We hope these responses address your concerns, and we are happy to clarify anything further.

Thanks for the detailed response. I think it clarifies most of my concerns and I maintain my previous Weak Accept rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.