- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Ex-Omni-2D, a dialogue model that answers a multimodal query with response text, personalized speech, and a talking-avatar video of a reference speaker. A Qwen3-8B LLM first writes a five-field Visual Thought Plan (VTP) covering scene, emotion, style, and motion, and then the response. A speech generator initialized from Qwen3-TTS-0.6B produces 16-codebook speech units at 12.5 Hz. A video generator built on Wan2.1-1.3B and OmniAvatar uses these same units, repeated to 25 FPS, along with the VTP and a reference latent. The authors say the shared unit interface lets the speech and video pathways train on separate data (speech dialogue data vs. about 140K SpeakerVid clips). A block-causal streaming variant, distilled with StreamAvatar/AnyFlow-style training and a reference sink, lowers time to first video chunk from 39.2 s to 6.6 s. The evaluation covers audio-video quality on 200 CommonEval queries (Table 1), OmniCharacter dialogue (Table 2), VoiceBench speech QA (Table 3), latency (Table 4), and ablations on the VTP and reference speech (Tables 5 to 7).
Critical strengths
- Using one set of speech units for both the audio and the video is a practical design. It gives the two outputs a shared frame-level timing signal, and it removes the need for paired dialogue-speech-video data, which is scarce. This is the paper’s main contribution and the reason the system can be trained at all.
- The paper is unusually candid about trade-offs. It reports where the method loses: lower SC/IQ, the VTP cost on BBH (Table 6), weak emotion control (Fig. 2b), and streaming degradation. That makes the results easier to trust.
Critical weaknesses
- The evidence that the VTP helps is weak. Removing it changes SC by about 1 point and Sync-C by 0.3 (Table 5), with no variance reported. It also costs 2.4 BBH points, and editing the emotion field often has no visible effect. The paper's central claim is that the LLM decides how the agent should appear in response to the dialogue. The appendix checks whether the video realizes the generated VTP (Table 14) and reports a human study on lip sync and facial naturalness (Table 18), but neither tests whether the planned behavior fits the dialogue, and neither compares against a neutral VTP. The motion-field intervention (Table 15) covers only 16 examples, and its confidence intervals include zero.
- The integrated design does not beat a simple cascade. Table 7 feeds Ex-Omni-2D’s upstream output into the off-the-shelf OmniAvatar renderer and gets SC 98.19 and Sync-C 5.48. That is better than Ex-Omni-2D’s own full-sequence video generator (94.62 and 4.95, Table 1), which undercuts the claim in Section 4 that the advantage comes from native coupling. The streaming variant drops further (SC 93.91, Sync-C 4.00), and first audio still takes about 6 s, which limits the interactive use case.
Questions for authors
- Can you provide a human or VLM-judge study showing that VTP-conditioned videos fit the dialogue context (emotion, gesture) better than videos generated with a neutral VTP? This is the main thing that could change my assessment.
- Given that the Table 7 cascade (Ex-Omni-2D upstream plus OmniAvatar) beats your own full-sequence generator on SC and Sync-C, what concrete benefit does the joint video generator provide? Was the cascade given the VTP or just the audio?
- Clarification: are the Table 5 differences consistent across seeds or sample subsets? Also, why is first audio output about 5.6 s when speech TTFC is 0.2 s?
2 comments
Response to Q3
(1) Each configuration is evaluated on 200 test samples with stochastic generation, and averaging across samples helps reduce the influence of random variation in individual outputs. We also checked the comparisons in Table 5 of the paper across three different seeds and observed consistent overall trends. (2) Generating the VTP before speech begins is a major source of startup delay. FAC includes this wait, whereas Speech TTFC measures only the speech-generation stage. This is a limitation of the current pipeline. With further vLLM-Omni inference optimization, our latest tests can complete the full VTP in under 1 s. Please see the Streaming section of our anonymous demo.
Response to Reviewer vR3L
Thank you for reconsidering our contribution and revising your rating from 2 to 3. Your reevaluation is very encouraging to us! We hope the point-by-point responses below address your remaining concerns, and we welcome any further questions or requests for clarification.
Response to W1
To test dialogue appropriateness directly, we compare trained VTPs with a fixed neutral plan on 200 requests, keeping the reply, speech, references, video generator, and seed fixed. Trained plans come from our VTP-supervised model; neutral is a fixed plan. Qwen2.5-VL-32B judges 12 chronological frames against the query and reply, with plans and method names hidden. We score trained wins/ties/losses as +1/0/−1 and average across both orders and requests. As shown in Table J, mean gesture preference favors trained guidance over neutral guidance, with a slightly higher mean expression preference (+0.003). Mean overall contextual preference is +0.165 in the same comparison. These results support a practical benefit from the model’s VTP.
Table J. Dialogue appropriateness of trained and neutral guidance.
Response to W2
(1) Our contribution is a video branch that consumes dialogue-generated VTPs and the same predicted speech units used for audio decoding, avoiding waveform re-encoding in the video-conditioning path. This shared interface supports training the dialogue/speech and video pathways on separate data sources and enables speech-aligned block-causal streaming. Although the unit-conditioned branch currently trails OmniAvatar on lip synchronization and facial naturalness (Table 18 in the paper appendix), the bridge provides a practical interface for adding reference-conditioned visual responses and speech-aligned streaming to a dialogue model. Under the matched deployment, streaming reduces first-video latency relative to our full-sequence model (Table 4 of the paper; Table 17 in the paper appendix).
(2) For startup latency, VTP-first planning contributes to the initial wait. With further vLLM-Omni optimization, our latest tests complete the full VTP in under 1 s. Please see the Streaming section of our anonymous demo.
Response to Q1
Following your suggestion, we keep the reply, speech, references, renderer, and seed fixed and compare the following plans on 200 requests: trained plans from our VTP-supervised model and a fixed neutral plan. Neutral is a fixed VTP, not removal of VTP conditioning. A frozen Qwen2.5-VL-32B judges 12 chronological frames against the query and reply, without plans or method names, in both video orders. We average trained wins/ties/losses (+1/0/−1) across orders and requests; positive scores favor trained plans. As shown in Table K, trained guidance has a mean gesture preference of +0.170 over neutral guidance. Mean overall contextual preference is +0.165 in the same comparison. Mean expression preference is slightly higher than neutral (+0.003). These gains provide direct evidence that our automatically generated VTPs improve dialogue-appropriate gestures and overall contextual preference over fixed neutral guidance. These scores assess dialogue appropriateness directly, complementing the original plan-alignment and motion diagnostics (Appendix F of the paper, Tables 14–15).
Table K. Dialogue appropriateness of trained and neutral guidance.
Response to Q2
(1) Our video branch directly consumes dialogue-generated VTPs and the same predicted speech units used for audio decoding, enabling speech-aligned streaming. This shared interface avoids waveform re-encoding and supports training with separate dialogue/speech and video datasets. These are practical integration benefits, despite the current quality gap to OmniAvatar (Tables 1 and 7 of the paper; Table 18 in the paper appendix).
(2) For the cascade in Table 7 of the paper, we supplied both generated audio and a separate text description. We provided both inputs to make full use of the baseline's audio- and text-conditioning capabilities and ensure a fair comparison.