acceptodds
Under review as a conference paper at ICLR 2027

Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence

Abstract

Omni-modal dialogue models can understand multimodal inputs and synthesize spoken replies, but a spoken answer still leaves the agent visually absent. We introduce Ex-Omni-2D, a framework that answers a multimodal query with coordinated text, personalized speech, and reference-conditioned video. The LLM first generates a structured Visual Thought Plan (VTP) followed by the response text. The speech generator then produces multi-codebook speech units from response-side representations. These speech units are decoded into audio and aligned with video frames, giving the speech and avatar modules a common timing signal while allowing them to learn from different data sources. The full-sequence video generator realizes the framework by conditioning on reference appearance, VTP semantics, and frame-aligned speech units. This is the configuration used for our main quality evaluation. For latency-sensitive settings, we additionally adapt it for few-step, block-causal streaming generation with a dedicated reference-image sink and speech-aligned chunks. This provides incremental output with lower startup latency than full-sequence decoding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Omni-modal dialogue models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vR3LOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Ex-Omni-2D, a dialogue model that answers a multimodal query with response text, personalized speech, and a talking-avatar video of a reference speaker. A Qwen3-8B LLM first writes a five-field Visual Thought Plan (VTP) covering scene, emotion, style, and motion, and then the response. A speech generator initialized from Qwen3-TTS-0.6B produces 16-codebook speech units at 12.5 Hz. A video generator built on Wan2.1-1.3B and OmniAvatar uses these same units, repeated to 25 FPS, along with the VTP and a reference latent. The authors say the shared unit interface lets the speech and video pathways train on separate data (speech dialogue data vs. about 140K SpeakerVid clips). A block-causal streaming variant, distilled with StreamAvatar/AnyFlow-style training and a reference sink, lowers time to first video chunk from 39.2 s to 6.6 s. The evaluation covers audio-video quality on 200 CommonEval queries (Table 1), OmniCharacter dialogue (Table 2), VoiceBench speech QA (Table 3), latency (Table 4), and ablations on the VTP and reference speech (Tables 5 to 7).

Critical strengths
  1. Using one set of speech units for both the audio and the video is a practical design. It gives the two outputs a shared frame-level timing signal, and it removes the need for paired dialogue-speech-video data, which is scarce. This is the paper’s main contribution and the reason the system can be trained at all.
  2. The paper is unusually candid about trade-offs. It reports where the method loses: lower SC/IQ, the VTP cost on BBH (Table 6), weak emotion control (Fig. 2b), and streaming degradation. That makes the results easier to trust.
Critical weaknesses
  1. The evidence that the VTP helps is weak. Removing it changes SC by about 1 point and Sync-C by 0.3 (Table 5), with no variance reported. It also costs 2.4 BBH points, and editing the emotion field often has no visible effect. The paper's central claim is that the LLM decides how the agent should appear in response to the dialogue. The appendix checks whether the video realizes the generated VTP (Table 14) and reports a human study on lip sync and facial naturalness (Table 18), but neither tests whether the planned behavior fits the dialogue, and neither compares against a neutral VTP. The motion-field intervention (Table 15) covers only 16 examples, and its confidence intervals include zero.
  2. The integrated design does not beat a simple cascade. Table 7 feeds Ex-Omni-2D’s upstream output into the off-the-shelf OmniAvatar renderer and gets SC 98.19 and Sync-C 5.48. That is better than Ex-Omni-2D’s own full-sequence video generator (94.62 and 4.95, Table 1), which undercuts the claim in Section 4 that the advantage comes from native coupling. The streaming variant drops further (SC 93.91, Sync-C 4.00), and first audio still takes about 6 s, which limits the interactive use case.
Questions for authors
  1. Can you provide a human or VLM-judge study showing that VTP-conditioned videos fit the dialogue context (emotion, gesture) better than videos generated with a neutral VTP? This is the main thing that could change my assessment.
  2. Given that the Table 7 cascade (Ex-Omni-2D upstream plus OmniAvatar) beats your own full-sequence generator on SC and Sync-C, what concrete benefit does the joint video generator provide? Was the cascade given the VTP or just the audio?
  3. Clarification: are the Table 5 differences consistent across seeds or sample subsets? Also, why is first audio output about 5.6 s when speech TTFC is 0.2 s?
2 comments
AuthorsOpenReview ↗

Response to Reviewer vR3L

Thank you for reconsidering our contribution and revising your rating from 2 to 3. Your reevaluation is very encouraging to us! We hope the point-by-point responses below address your remaining concerns, and we welcome any further questions or requests for clarification.

Response to W1

To test dialogue appropriateness directly, we compare trained VTPs with a fixed neutral plan on 200 requests, keeping the reply, speech, references, video generator, and seed fixed. Trained plans come from our VTP-supervised model; neutral is a fixed plan. Qwen2.5-VL-32B judges 12 chronological frames against the query and reply, with plans and method names hidden. We score trained wins/ties/losses as +1/0/−1 and average across both orders and requests. As shown in Table J, mean gesture preference favors trained guidance over neutral guidance, with a slightly higher mean expression preference (+0.003). Mean overall contextual preference is +0.165 in the same comparison. These results support a practical benefit from the model’s VTP.

Table J. Dialogue appropriateness of trained and neutral guidance.

ComparisonScored pairsMean expression preferenceMean gesture preference
Trained − neutral200/200+0.003+0.170

Response to W2

(1) Our contribution is a video branch that consumes dialogue-generated VTPs and the same predicted speech units used for audio decoding, avoiding waveform re-encoding in the video-conditioning path. This shared interface supports training the dialogue/speech and video pathways on separate data sources and enables speech-aligned block-causal streaming. Although the unit-conditioned branch currently trails OmniAvatar on lip synchronization and facial naturalness (Table 18 in the paper appendix), the bridge provides a practical interface for adding reference-conditioned visual responses and speech-aligned streaming to a dialogue model. Under the matched deployment, streaming reduces first-video latency relative to our full-sequence model (Table 4 of the paper; Table 17 in the paper appendix).

(2) For startup latency, VTP-first planning contributes to the initial wait. With further vLLM-Omni optimization, our latest tests complete the full VTP in under 1 s. Please see the Streaming section of our anonymous demo.

Response to Q1

Following your suggestion, we keep the reply, speech, references, renderer, and seed fixed and compare the following plans on 200 requests: trained plans from our VTP-supervised model and a fixed neutral plan. Neutral is a fixed VTP, not removal of VTP conditioning. A frozen Qwen2.5-VL-32B judges 12 chronological frames against the query and reply, without plans or method names, in both video orders. We average trained wins/ties/losses (+1/0/−1) across orders and requests; positive scores favor trained plans. As shown in Table K, trained guidance has a mean gesture preference of +0.170 over neutral guidance. Mean overall contextual preference is +0.165 in the same comparison. Mean expression preference is slightly higher than neutral (+0.003). These gains provide direct evidence that our automatically generated VTPs improve dialogue-appropriate gestures and overall contextual preference over fixed neutral guidance. These scores assess dialogue appropriateness directly, complementing the original plan-alignment and motion diagnostics (Appendix F of the paper, Tables 14–15).

Table K. Dialogue appropriateness of trained and neutral guidance.

ComparisonMean expression preferenceMean gesture preference
Trained − neutral+0.003+0.170

Response to Q2

(1) Our video branch directly consumes dialogue-generated VTPs and the same predicted speech units used for audio decoding, enabling speech-aligned streaming. This shared interface avoids waveform re-encoding and supports training with separate dialogue/speech and video datasets. These are practical integration benefits, despite the current quality gap to OmniAvatar (Tables 1 and 7 of the paper; Table 18 in the paper appendix).

(2) For the cascade in Table 7 of the paper, we supplied both generated audio and a separate text description. We provided both inputs to make full use of the baseline's audio- and text-conditioning capabilities and ensure a fair comparison.

Response to Q3

(1) Each configuration is evaluated on 200 test samples with stochastic generation, and averaging across samples helps reduce the influence of random variation in individual outputs. We also checked the comparisons in Table 5 of the paper across three different seeds and observed consistent overall trends. (2) Generating the VTP before speech begins is a major source of startup delay. FAC includes this wait, whereas Speech TTFC measures only the speech-generation stage. This is a limitation of the current pipeline. With further vLLM-Omni inference optimization, our latest tests can complete the full VTP in under 1 s. Please see the Streaming section of our anonymous demo.

Reviewer_opbnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Ex-Omni-2D produces conversational text, personalized speech, and expressive avatar videos within a unified framework. It uses a structured Visual Thought Plan (VTP) to guide generation at the semantic level, while shared multi-codebook speech units bridge speech and video generation. Trained on separate dialogue and audiovisual datasets, the framework supports both full-sequence and streaming generation. Its evaluation examines conversational and audiovisual quality, latency, the contributions of individual components through ablation studies, and human preferences.

Critical strengths
  • Structured visual plans and shared speech units establish explicit connections among dialogue, speech, and video generation. Training with separate sources of supervision reduces reliance on fully paired conversational response videos, improving the framework’s practical applicability.
  • The paper presents both full-sequence and streaming variants, supported by component ablations and human evaluations. Under the reported eight-GPU configuration, streaming reduces the time to the first available video output from 39.19 to 6.64 seconds, representing a substantial engineering improvement.
Critical weaknesses
  • The methodological novelty is insufficiently established. The design overlaps with X-Streamer, EmpathyEar, and OmniHuman-1.5, yet these connections are insufficiently discussed. Without a matched LLM-speech-avatar baseline using comparable visual guidance, the added value of the proposed interfaces remains unclear.
  • Dialogue-appropriate expressiveness remains insufficiently demonstrated. Video-plan alignment does not establish contextual appropriateness, while human evaluation assesses only lip synchronization and facial naturalness. The motion intervention shows a promising trend, but uses just 16 examples, with both mean-effect confidence intervals including zero. Evidence for the central expressive-control claim is therefore limited.
  • The proposed method does not achieve the strongest performance among the evaluated baselines on several key metrics. OmniAvatar achieves better audiovisual synchronization and is preferred by human raters for lip synchronization and facial naturalness, while Qwen2.5-Omni scores higher on speech QA. These gaps may reflect trade-offs in integrated dialogue-to-video generation, but the paper does not clearly establish that visual planning offers enough benefit to offset the quality loss.
Questions for authors
  • Under matched response text, reference inputs, and video generator, how does the trained VTP compare with visual guidance produced by a prompted LLM?
  • What evidence demonstrates improved dialogue-appropriate expression, beyond plan-video agreement and increased motion magnitude?
5 comments
AuthorsOpenReview ↗

Response to Reviewer opbn

We appreciate your constructive feedback. Below, we respond to the points you raised. Please let us know if any point would benefit from further clarification.

Response to W1

We integrate visual planning with dialogue generation and use the same predicted speech units for audio decoding and incremental video. (1) Compared with EmpathyEar, we use a different interface to condition video generation. EmpathyEar learns an emotion/scene/reply/profile meta-response and uses StyleTTS2 for speech and EAT for talking faces. Our video model receives explicit scene, affect, movement-style, and motion descriptions together with the units used for speech decoding, and supports a streaming adaptation. (2) Compared with OmniHuman-1.5, we generate the visual plan and the reply within the same dialogue model. OmniHuman-1.5 uses an Analyzer–Planner pipeline with supplied audio, a reference image, and optional text. Our dialogue model generates the visual plan and then the user-facing reply from the query; the speech generator predicts the shared acoustic units. (3) Compared with X-Streamer, we use explicit VTP text and shared acoustic units as inputs to the video model. X-Streamer freezes its GLM-4-Voice Thinker and conditions the video Actor on text–audio hidden states. We train the dialogue model to emit VTP–response text and the video branch to consume VTP text and multi-codebook units. Both systems stream, but the conditioning representations and training designs differ. We will make these distinctions and shared ideas explicit in the Related Work section.

Response to W2

To address this concern, we compare trained VTPs with a fixed neutral plan on 200 requests, keeping the reply, speech, references, renderer, and seed fixed. Qwen2.5-VL-32B judges 12 chronological frames against the query and reply, with plans and method names hidden. We score trained wins/ties/losses as +1/0/−1 and average across both orders and requests. As shown in Table E, trained VTPs have a mean gesture preference of +0.170 over fixed neutral guidance. Mean overall contextual preference is +0.165. These gains provide direct evidence that our automatically generated VTPs improve dialogue-appropriate gestures and overall contextual preference over fixed neutral guidance. Mean expression preference is slightly higher than neutral (+0.003).

Table E. Dialogue appropriateness of trained and neutral guidance.

ComparisonMean expression preferenceMean gesture preference
Trained − neutral+0.003+0.170

Response to W3

(1) The added benefits are visual guidance, direct shared-unit conditioning, and incremental audiovisual output. Our video training makes the predicted speech units directly usable for video and block-causal streaming.

(2) Following your suggestion, we add a comparison of our video branch with OmniAvatar using the same trained VTP, reference inputs, and generated speech. Our branch takes predicted speech units; OmniAvatar takes their decoded waveform. Table F reports paired differences (ours − OmniAvatar). Positive values favor ours. As shown in Table F, mean contextual and gesture preferences favor ours, while OmniAvatar scores better on synchronization and sampled-frame naturalness.

Table F. Paired renderer comparison under identical VTP guidance.

Metric, same trained VTPMean difference (ours − OmniAvatar)
Sync-C−1.239
Overall contextual preference+0.026
Gesture preference+0.062
Sampled-frame naturalness preference−0.049

(3) Additionally, we compare trained VTPs with a fixed neutral plan, keeping the reply, speech, references, renderer, and seed fixed. Qwen2.5-VL-32B judges 12 chronological frames against the query and reply, with plans and method names hidden. We score trained wins/ties/losses as +1/0/−1 and average across both orders and requests. As shown in Table G, mean gesture preference favors trained guidance over neutral guidance, with a slightly higher mean expression preference (+0.003). Streaming also reduces first-video latency relative to our full-sequence model (Table 4 of the paper; Table 17 in the paper appendix). For speech QA, Ex-Omni-2D ranks second behind Qwen2.5-Omni on all three benchmarks (Table 3 of the paper) while also supporting visual planning, personalized speech, and video responses.

Table G. Trained versus neutral visual guidance.

ComparisonMean expression preferenceMean gesture preference
Trained − neutral+0.003+0.170

Response to Q1

Following your suggestion, we compare trained VTPs with plans generated by prompting Qwen3-Omni-30B-A3B with the query, fixed reply, and reference image, keeping the reply, speech, references, video generator, and sampling settings fixed. A blinded Qwen2.5-VL-32B judges 12 chronological frames against the query and reply. Scores average trained wins/ties/losses (+1/0/−1) across both presentation orders and requests. As shown in Table H, trained plans have a slightly higher mean gesture preference (+0.036) and a lower mean expression preference (−0.023) than prompted plans. Our practical advantage is automatic VTP generation within the dialogue model, without a separate planning model at inference.

Table H. Trained versus prompted visual guidance.

ComparisonMean expression preferenceMean gesture preference
Trained − prompted−0.023+0.036

Response to Q2

To test dialogue appropriateness directly, we compare trained VTPs with a fixed neutral plan on 200 test samples, keeping the reply, speech, references, video generator, and seed fixed. Trained plans come from our VTP-supervised model; neutral is a fixed plan. Qwen2.5-VL-32B judges 12 chronological frames against the query and reply, with plans and method names hidden. We score trained wins/ties/losses as +1/0/−1 and average across both orders and requests. As shown in Table I, mean gesture preference favors trained guidance over neutral guidance (+0.170), with a slightly higher mean expression preference (+0.003). Mean overall contextual preference is +0.165 in the same comparison. These results show a practical benefit of our approach: the dialogue model's automatically generated VTPs produce more dialogue-appropriate gestures and higher overall contextual preference than fixed neutral guidance. This evaluation directly judges generated behavior against the query and reply, complementing plan-alignment and motion diagnostics (Appendix F of the paper, Tables 14–15).

Table I. Dialogue appropriateness of trained and neutral guidance.

ComparisonMean expression preferenceMean gesture preference
Trained − neutral+0.003+0.170

Dear Reviewer opbn,

We appreciate your suggestions on clarifying our contribution and evaluating dialogue appropriateness.

  • Relation to Prior Work. We clarified how Ex-Omni-2D differs from EmpathyEar, OmniHuman-1.5, and X-Streamer (see Response to W1).
  • Dialogue Appropriateness. We added a blinded VLM study on test sets and found higher gesture and contextual preference with trained VTPs than with neutral guidance (see Responses to W2 and Q2).
  • Trade-offs. We added a matched OmniAvatar comparison and discussed the trade-offs (see Response to W3).
  • Trained Versus Prompted Plans. We found that trained VTPs scored slightly higher on gestures and slightly lower on expressions than plans generated by prompting Qwen3-Omni under matched conditions (see Response to Q1).

We would appreciate any feedback you have time to share before the discussion closes, especially on whether the responses address your questions.

Best regards,

The Authors

Reviewer_TSZcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends omni-modal dialogue models so that they also produce a reference-conditioned talking-avatar video. Given a text or speech query, a reference image and a reference voice, Ex-Omni-2D generates response text, personalized speech and a synchronized avatar video. The LLM (a Qwen3-8B backbone with Qwen3-VL vision and speech encoders) first emits a structured Visual Thought Plan (VTP). The VTP has five textual fields (first-frame scene, scene, emotion, movement style, motion description) inside a constrained <thinking> block, followed by the user-facing response. A Qwen3-TTS-0.6B-initialized speech generator produces 16-codebook units at 12.5 Hz from the response-span hidden states and the reference voice. The same units serve as the video's audio condition: embedded, summed over codebooks, and repeated twice per acoustic frame to align with 25 FPS. A Wan2.1-1.3B video generator initialized from OmniAvatar is conditioned on the reference latent, the VTP text and the frame-aligned units. Training is decoupled across data sources. Stage 1 aligns speech using ASR/TTS data. Stage 2 adapts dialogue on InstructS2S and OmniCharacter, with VTPs pseudo-labeled by Qwen3-VL-235B. Stage 3 trains the video model on 140K filtered SpeakerVid clips, with VTPs annotated from the clips and teacher-forced units. Stage 4 distills a block-causal, 8-step streaming student with a reference-image KV sink. Experiments report audio quality, VBench video metrics, Sync-C, VoiceBench QA, OmniCharacter dialogue scores, latency and several VTP ablations. The full system does not lead on most rendering metrics, but it is the only system that covers the full dialogue-to-video pipeline, and streaming reduces the first-video latency from 39 s to 6.6 s.

Critical strengths

1. The problem setting is well motivated. Letting the dialogue model decide both what to say and how the agent should appear is a natural next step for omni-modal assistants, and the paper states it clearly.

2. The data-decoupling design is practical. Because the VTP and the speech units act as interfaces, dialogue, speech and avatar modules can learn from separate corpora without paired dialogue–speech–video data, which do not exist at scale.

3. The empirical reporting is unusually honest. The authors state that the system does not dominate the rendering baselines, that VTP supervision costs QA accuracy (Table 6), that waveform conditioning gives better sync (Table 16), and that emotion control can fail. The reference-sink ablation includes paired bootstrap confidence intervals.

4. The streaming adaptation is engineered carefully (clean-cache writes only, a dedicated reference sink with an offset RoPE position, rate-aligned 6-unit / 12-frame chunks) and documented in enough detail to reproduce.

Critical weaknesses

1. The central claim, that the LLM plans visual behavior appropriate to the dialogue, is not validated. All evidence for the VTP is indirect or weak. Removing the response-specific VTP changes SC by 1.04 and Sync-C by 0.30 (Table 5). Neither metric measures semantic appropriateness, as the paper itself notes. The motion-field intervention uses 16 samples, and its bootstrap confidence intervals include zero (Table 15). Emotion edits "may fail to emerge". What is missing is a human or judge evaluation of whether the generated affect and motion fit the dialogue content, comparing the generated VTP, a neutral VTP and a shuffled VTP from another query.

2. The VTP seen in training differs from the VTP produced at inference. In Stage 3, VTPs are descriptive captions produced by Qwen3-VL-235B while watching the target clip. At inference, VTPs are prescriptive plans that the LLM learned from Stage-2 pseudo-labels created without any video. The video generator is therefore trained to follow observed-behavior captions and tested on imagined plans. The 43% first-frame grounding score (Table 14) suggests the gap matters. A distributional comparison of the two VTP sources (field lengths, vocabulary, emotion-label distribution) and a cross-condition test are needed.

3. The rationale for the shared speech-unit interface is internally inconsistent. Waveform + wav2vec conditioning gives higher Sync-C (5.83 vs 4.95, Table 16), and the units are justified by a 40 ms lower conditioning latency, while time to first audio is about 5.6–6.0 s (Table 4). The data-decoupling argument applies equally to waveform features, since SpeakerVid clips have real audio. The video generator also starts from OmniAvatar, whose Sync-C (5.64) is higher than the final model's (4.95), so switching to unit conditioning appears to degrade the very capability the system inherits. The paper should justify the interface on grounds other than latency, for example tighter acoustic–visual consistency, or adopt the better-performing option.

4. A key controlled baseline is missing. The system is a cascade linked by text and units (LLM → TTS → video), so "native" coupling is mainly that the LLM emits a caption. The decisive comparison is the same Ex-Omni-2D LLM emitting the VTP, passed as a text prompt to OmniAvatar or UniAVGen (both accept text prompts), together with the generated waveform. Table 7 changes only the upstream source and not the prompt pathway, so it does not test whether a jointly trained, VTP-consuming generator beats a strong prompt-driven cascade.

5. The prosody of the speech is not conditioned on the plan. The speech generator conditions on response-span hidden states and the reference voice, but not on the VTP. Facial emotion (from the VTP) and vocal affect (from the response text) are therefore decided independently and may conflict, for example an "excited" face with neutral prosody. This weakens the claim of coordinated multimodal responses and is not evaluated.

6. The video evaluation is narrow and partly unsuitable. Only 200 CommonEval queries are used, with references taken from the SpeakerVid test split (the same source domain as training). The paper reports no identity preservation against the reference image (e.g., ArcFace similarity), no distribution-level metric (FVD), and DD, which the authors admit does not measure quality. In the human study (Table 18), Ex-Omni-2D ranks last on lip sync and below OmniAvatar on naturalness. The overall evidence that the proposed generator renders better than its own initialization is weak.

7. The dialogue comparison favors the proposed model. OmniCharacter is part of the Stage-2 training data, yet Table 2 compares against baselines that were not trained on it. Even so, the average gain over Qwen3-8B is 0.019, with large drops in Knowledge-Exposure (2.722 → 2.076) and Persona-Behavior (3.565 → 2.911). "Highest fluency, coherency and consistency" may reflect in-domain style matching more than better dialogue ability.

8. Several protocol details are inconsistent. Table 10 lists text/audio CFG of 3.5/8.5 for the main full-sequence results, Table 17 states CFG 4, and Table 19 shows 4/4 to be best for lip sync and naturalness. Please clarify which setting produced Table 1 and why the main configuration is not the human-preferred one. The section header "Human Preference across Guidance Scales" in Appendix F is also empty.

Questions for authors

1. With speech TTFC at about 0.2 s, what makes up the roughly 5.6 s FAC? Is it VTP generation before the response, codec decoding, or pipeline scheduling? This determines whether the VTP-first protocol fundamentally delays audio.

2. How often does the generated VTP contradict the response content (e.g., a "sad" emotion for a cheerful reply)? Was any consistency filter applied to the Stage-2 pseudo-labels from Qwen3-VL-235B?

3. In streaming mode, the VTP is encoded once for the whole response. Can the plan vary over time, or must a multi-sentence response with changing affect be described by one static five-field plan?

4. Why is the first-frame field generated by the LLM at all when the first frame is the supplied reference image? Does conditioning the video model on an inaccurate first-frame description (43% grounded) hurt quality?

5 comments
AuthorsOpenReview ↗

Response to Reviewer TSZc

Thank you for your detailed review. We address your concerns and questions point by point below. We welcome any follow-up questions or requests for clarification.

Response to W1

Following your suggestion, we compare three VTP settings on 200 requests while keeping the reply, speech, references, renderer, and seed fixed: trained, generated by our VTP-supervised model; neutral, a fixed neutral plan; shuffled, emotion/style/motion fields from another request with the current first-frame and scene fields retained. Qwen2.5-VL-32B judges 12 chronological frames against the query and reply, without plans or method names, in both video orders. We score trained wins/ties/losses as +1/0/−1, average the two orders, then average across requests. Positive scores favor trained plans. As shown in Table A, trained plans have a mean gesture preference of +0.170 over neutral guidance. Mean overall contextual preference is +0.165. These gains provide direct evidence that our automatically generated VTPs improve dialogue-appropriate gestures and overall contextual preference over fixed neutral guidance. These scores assess dialogue appropriateness directly, complementing the original plan-alignment and motion diagnostics (Appendix F of the paper, Tables 14–15). Mean expression preference is slightly higher than neutral (+0.003). Shuffled plans have higher mean expression and gesture preferences in this test. We will state these limits in the revised paper's Limitations section.

Table A. Dialogue appropriateness of expressions and gestures across VTP settings.

ComparisonMean expression preferenceMean gesture preference
Trained − neutral+0.003+0.170
Trained − shuffled−0.033−0.105

Response to W2

We compare the mean token length of each VTP field in 1,000 video-training descriptions and 200 generated test plans. As shown in Table B, video-training descriptions and generated test plans differ in length. Our design integrates visual planning into dialogue generation, without requiring a separate planning model at test time. In a separate replacement test, we replaced the entire dialogue-generated VTP with descriptions produced using the video-training annotation procedure (Appendix D of the paper). On the same 200 requests, with all other generation conditions fixed, Sync-C increased from 4.95 to 5.08. This illustrates a trade-off: an additional VTP-generation step can improve synchronization, while our default produces the VTP within the dialogue model itself. We will discuss this trade-off and the remaining caption–plan gap in Limitations.

Table B. Mean VTP field lengths in tokens.

VTP fieldTraining descriptionsGenerated test plans
first_frame_scene62.0555.69
scene43.5928.10
motion_description52.7436.34

Response to W3

(1) We use speech units because our speech model already predicts them, and the same units can directly drive both audio decoding and video generation. This removes waveform conversion and re-encoding from the video-conditioning path. Waveform features can also support heterogeneous supervision.

(2) The paper reports both conditioning overhead and synchronization quality. The 16-codebook interface takes 0.011 s to prepare video conditioning, compared with 0.051 s for waveform conditioning, while Sync-C is 4.95 versus 5.83. It also improves on the single-codebook Sync-C of 2.07 (Table 16 of the paper). The shared-unit interface therefore offers lower conditioning overhead than waveform conditioning and better synchronization than the single-codebook alternative. We will explicitly discuss the remaining synchronization gap in the revised paper's Limitations section. Time to first audio also includes VTP planning. With vLLM-Omni optimization, our latest tests complete the full VTP in under 1 s; see the Streaming section of our anonymous demo.

(3) The shared units also provide the timing for streaming: six units at 12.5 Hz form a 0.48-second block aligned with 12 video frames at 25 FPS.

Response to W4

(1) We train the video branch to generate video directly from the speech model's predicted units and support speech-aligned streaming. Stage 3 learns VTP- and unit-conditioned generation from video clips, followed by block-causal streaming adaptation. This gives audio and video a shared acoustic timing signal and removes waveform conversion and re-encoding from the video-conditioning path. The resulting streaming configuration can further reduce first-video latency relative to our full-sequence model (Sec. 4 of the paper; Table 4 of the paper).

(2) Following your suggestion, we add a comparison of our video branch with OmniAvatar using the same trained VTP, reference image, and underlying speech. Our branch receives predicted speech units. OmniAvatar receives their decoded waveform. We report paired differences (ours − OmniAvatar); positive values favor ours. As shown in Table C, mean contextual and gesture preferences favor ours, while OmniAvatar scores better on synchronization and sampled-frame naturalness. The paper already reports and discusses synchronization and rendering-quality limits (Tables 16 and 18 of the paper; Appendix G of the paper). We will clarify the contribution in the revised Sec. 4 of the paper.

Table C. Paired renderer comparison under identical VTP guidance.

Metric, same trained VTPMean difference (ours − OmniAvatar)
Sync-C−1.239
Overall contextual preference+0.026
Gesture preference+0.062
Sampled-frame naturalness preference−0.049

Response to W5

The speech pathway has indirect access to the VTP through the reply representations. The LLM generates the plan before the reply in one causal sequence, so the reply hidden states can attend to the preceding plan. These states, together with the reply tokens and reference voice, condition speech generation. The video generator then uses both the VTP and the generated speech units. These connections provide a path for coordination between speech and facial behavior. When VTP semantics and speech-derived acoustic cues disagree, the video generator may balance the two. Our qualitative inspection found two apparent VTP–speech affective mismatches among 200 test examples. In both cases, the VTP specified sadness or neutral affect while the speech sounded positive or cheerful. We will expand this discussion in the revised paper.

Response to W6

(1) To assess appearance preservation, we compare configurations with and without the reference sink on 99 matched examples at streaming chunk 16. Retaining the reference sink improves DINO similarity to the reference image from 0.6862 to 0.7625 (Fig. 3 in the paper appendix). This supports the reference sink as a practical mechanism for retaining appearance during streaming.

(2) Our contribution combines dialogue-conditioned visual planning, a shared speech–video interface, and speech-aligned streaming generation. OmniAvatar remains preferred by human raters for lip synchronization and facial naturalness (Table 18 in the paper appendix). We will make the evaluation scope and remaining identity and rendering limitations explicit in the revised Limitations, building on the existing discussion of drift and streaming quality in Appendix G of the paper.

Response to W7

(1) Regarding comparison fairness, Stage 2 uses OmniCharacter's official training split, and all models are evaluated on the same official test examples under the benchmark protocol.

(2) Regarding the KE/PB declines, we consider it normal for a model covering dialogue, speech, and video tasks to improve on some metrics and decline on others. As a separate test of dialogue-context use, with the model and Role Card fixed, adding preceding dialogue turns improves all twelve dimensions and raises the average from 2.769 to 3.283 (Table 12 in the paper appendix). This controlled gain shows that the model benefits from dialogue history across all twelve evaluated dimensions.

Response to W8

(1) For the settings, the main full-sequence results in Table 1 of the paper use text CFG 3.5 and audio CFG 8.5 (Table 10 in the paper appendix). For the full-sequence latency experiment, “CFG 4” means text CFG = audio CFG = 4, i.e., 4/4 (Table 17 iin the paper appendix). We will write both guidance scales explicitly in each protocol. We chose 3.5/8.5 for the main full-sequence quality results because stronger audio guidance improved automatic benchmark performance in our experiments. Human evaluation favored 4/4 for lip synchronization and facial naturalness (Table 19 in the paper appendix). This difference highlights a gap between automatic benchmark scores and human preference. We will explain this trade-off and the selection rationale in the revised paper.

(2) Table 19 in the paper appendix reports the human evaluation of full-sequence guidance scales. We will place it beside the “Human Preference across Guidance Scales” heading and add the following brief analysis.

"Human preference does not improve uniformly with guidance strength. Text/audio CFG 4/4 gives the best reported lip-synchronization and facial-naturalness ratings, while 8/8 gives a slightly better mean gesture rank but lower ratings on the other two criteria. Stronger audio guidance benefited our automatic benchmark scores, motivating the 3.5/8.5 setting for the main full-sequence results. This difference illustrates that the guidance setting preferred by automatic metrics may differ from that preferred by humans."

Response to Q1

Generating the VTP before speech begins is a major source of startup delay. FAC includes this wait, whereas Speech TTFC measures only the speech-generation stage. This is a limitation of the current pipeline. With further vLLM-Omni inference optimization, our latest tests can complete the full VTP in under 1 s. Please see the Streaming section of our anonymous demo.

Response to Q2

(1) We cross-checked the Stage-2 VTP labels from Qwen3-VL-235B with Qwen3-Omni and removed examples judged inconsistent with their associated responses. This cross-check provides a consistency safeguard for the training labels.

(2) Our qualitative inspection of the 200 test examples found two apparent VTP–speech affective mismatches (2/200, 1%). In both cases, the VTP specified sadness or neutral affect, while the generated speech sounded positive or cheerful. These observations suggest that obvious affective conflicts are infrequent in this test set, although they can still occur. We will discuss this limitation in the revised paper.

Response to Q3

The VTP supports describing emotion transitions within a response (Appendix D of the paper). The plan is encoded once for the response, without sentence-level timestamps or chunkwise replanning, so transition timing is not precisely controlled.

Response to Q4

We include the first-frame description to make appearance and scene cues explicit in text, complementing the reference-image latent. To test its effect, we used Qwen3-8B to make small changes only to this field on CommonEval, such as changing the stated clothing color. We kept the other VTP fields, reference inputs, generated speech, seed, and sampling settings fixed. As shown in Table D, the observed Sync-C decrease is 0.18. This controlled result indicates that the first-frame text affects synchronization even when the reference image is fixed. The effect on Sync-C is modest in this test.

Table D. Effect of first-frame description perturbations on Sync-C.

First-frame descriptionSync-C ↑
Original generated description4.95
Description perturbed by Qwen3-8B4.77

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.