acceptodds
Under review as a conference paper at ICLR 2027

Ex-Omni: Enabling 3D Facial Animation Generation for Omni-modal Large Language Models

18 views

Abstract

Omni-modal large language models (OLLMs) aim to unify multimodal understanding and generation, yet extending them to jointly produce speech and 3D facial animation remains largely underexplored. A key challenge is the mismatch between the discrete semantic reasoning of LLMs and the dense temporal dynamics required for 3D facial motion. We propose Expressive Omni (Ex-Omni), a framework that augments OLLMs with speech-accompanied 3D facial animation. Ex-Omni decouples semantic reasoning from temporal generation through a speech-unit generator with blendshape co-supervision and a non-autoregressive blendshape decoder, where speech units provide temporal scaffolding and hidden speech representations carry facially relevant cues. We further introduce a token-as-query gated fusion (TQGF) interface for controlled semantic injection, as well as InstructS2SF-1200K, a 1.2M-sample weakly supervised dataset for speech-accompanied facial animation. Extensive experiments show that Ex-Omni retains competitive speech QA capability while natively generating coordinated text, speech, and 3D facial animation, and approaches the Audio2Face-3D teacher cascade in synchronization and human preference.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Omni-modal large language models, 3D facial animation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_enYoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends an omni-modal LLM to generate speech-synchronized 3D facial animation (ARKit-52 blendshapes) along with text and speech. Ex-Omni is built on Qwen3-8B with a Whisper encoder. A Qwen3-0.6B speech-unit generator, in the style of GLM-4-Voice, produces discrete speech units conditioned on response tokens and LLM hidden states. A non-autoregressive blendshape decoder uses the unit embeddings, interpolated to frame rate, as queries and the unit generator's hidden states as context. Both connections use a gated cross-attention block (TQGF). Training data (InstructS2SF-1200K) is fully synthetic:

  • text from Emilia and InstructS2S is voiced by Qwen3-TTS with a single speaker;
  • NVIDIA Audio2Face-3D, conditioned on Audio2Emotion, produces blendshape pseudo-labels.

Ex-Omni beats OLLM + EmoTalk/UniTalker cascades on SyncNet, comes second to the Audio2Face-3D teacher cascade in SyncNet and human preference (41.8% vs. 55.8%), and keeps most of its speech-QA ability

Critical strengths
  1. Adding facial animation as an output of a speech-capable LLM is a practically relevant direction for avatars and embodied agents. Using speech units as a temporal scaffold for the face decoder is a sensible design.
  2. it is great that they explain why LVE against teacher outputs would be circular, add a controlled comparison with fixed upstream speech (Table 3), and openly include the teacher cascade as a stronger reference.
  3. They checked multiple evaluations including VoiceBench QA, UTMOS, speech–text WER, and text-only retention against the base Qwen3-8B. Also they report latency.
  4. The code, data pipeline, and 20 randomly sampled supplementary videos aid reproducibility and qualitative judgment.
Critical weaknesses
  1. The native face branch is a distilled student that doesn't beat its teacher, and the benefit of going native is not shown. In every face metric the teacher cascade (Ex-Omni speech → Audio2Face-3D) is better: Sync-D 8.92 vs. 9.21 and human preference 55.8% vs. 41.8% (Tables 3–4). So the practical alternative, running an OLLM and piping its speech into Audio2Face-3D, gives better faces today. The paper's argument for integration is avoiding an external S2F model at inference, but it never reports the latency or compute of the teacher cascade. The limitations section also says the overall real-time factor is above 1.0 and the latency gain is limited to the face branch. To justify the "native" framing, the paper needs to show some advantage the cascade cannot have. Examples: lower end-to-end latency or first-frame time, streaming, or facial behavior that depends on dialogue semantics rather than audio alone. Otherwise the contribution reduces to distilling Audio2Face-3D into an OLLM head.

  2. "Expressive" is not demonstrated. The introduction motivates the work with non-verbal cues in face-to-face interaction. But Sec. 1 redefines "expressive" as articulation and lip-sync. Appendix A notes that most pseudo-labels in the dialogue data are close to neutral, and the facial labels are a function of the audio only. The model therefore has no training signal linking dialogue content or intent to facial expression, which is what an LLM-integrated face model could uniquely offer. Nothing is evaluated for emotion, expression diversity, or alignment between text semantics and face. The name and motivation promise more than the method delivers.

  3. The evaluation is weak for 3D facial animation.

    • SyncNet is trained on real 2D talking faces and is applied here to renders of one ARKit template (Claire, the Audio2Face template). EmoTalk and UniTalker blendshape outputs may follow different conventions and scales than Audio2Face-3D's ARKit rig. Rendering everything on the teacher's native template may favor the teacher and its student independently of motion quality.
    • The temporal diagnostics (Table 12) show Ex-Omni's acceleration is about 3× the teacher's (Acc-L2 0.132 vs. 0.043). The paper describes this as "greater frame-to-frame variation," but high acceleration with a plain frame-wise MSE loss and no smoothness term usually means jitter. This needs a direct check, such as a jitter metric, a human rating of smoothness, or the supplementary videos at full frame rate.
    • No evaluation uses real captured facial motion. One option: tokenize real speech from a mocap set (e.g., the 3D-ETF/RAVDESS-based sets used by EmoTalk) into units and compare against ground truth. That would show whether the student learned facial motion or Audio2Face-3D's specific style.
Questions for authors
  1. Can you show any facial behavior that depends on dialogue content beyond the audio, which a speech-to-face cascade could not produce?
  2. Were all baselines' blendshape outputs mapped onto the same ARKit convention and scale before rendering? How do SyncNet results change on a neutral template not used by Audio2Face-3D?
  3. Is the higher acceleration in Table 12 jitter? Could you report a jitter or smoothness metric, or a human smoothness rating?
  4. How does the Direct LLM-to-Face variant perform when trained to convergence? What happens if you stop gradients from the face decoder into the unit generator?
6 comments
AuthorsOpenReview ↗

Response to Reviewer enYo

We really thank you for the careful evaluation and constructive suggestions! Below, we clarify the benefits of native generation and present additional evaluations.

Response to W1

The practical benefit of native generation is that one dialogue model generates text, speech, and facial animation without deploying separate Audio2Emotion and Audio2Face-3D models. Our main contribution is to give a multimodal dialogue model this capability within its own generation process. Since submission, we have further optimized our implementation on vLLM-Omni and measured the time from text input to a finished audio-visual response using this optimized implementation. As shown in Table J, Ex-Omni reduces median total generation time by 3.0%–14.3% relative to the Audio2Face-3D cascade across the tested response lengths. Complete-response generation takes less time than playback for all tested lengths, improving on the earlier implementation. We will update the earlier RTF statement in the paper's Limitations section to reflect these results.

Table J. Additional latency analysis

Output duration (s)Ex-Omni latency (s) ↓Ex-Omni + Audio2Face-3D latency (s) ↓
4.0753.0223.115
12.2377.4208.660
28.87415.76117.239
53.83630.51132.509

Response to Q1/W2

(1) We keep the response text, speech units, and audio unchanged, and change only the dialogue context. In some comparisons, facial coefficients change in the expected direction. However, the changes are small. This suggests that dialogue context can affect facial generation.

(2) We include natural facial behavior and speech–face coordination, as well as emotional expression. Our aim is to give an omni-modal dialogue model a human-like visual presence alongside its text and speech. The human study in Table 4 in the paper evaluates synchronization and naturalness jointly (mentioned in Appendix A, “Human Evaluation Protocol,” of the paper) and supports overall perceptual quality relative to the non-teacher cascades; the specialized teacher remains stronger. The study does not independently assess emotional expression or upper-face naturalness. Figure 2, Case 2 in the paper illustrates natural mouth closure after speech ends.

(3) In our additional CREMA-D evaluation, as shown in Table K, HSEmotion classifies 9/30 happy-input outputs as happy, while overall agreement with the intended emotion is only 37/179 (20.7%); predictions are predominantly neutral or happy. This is exploratory automatic recognition on rendered faces, not validated human judgment. These classifier-based results suggest some smile-like expression but do not establish reliable emotion matching or broad emotional diversity. Our training data focus on everyday conversations rather than emotionally diverse role-play, and most QA-style responses are close to neutral under our data-construction pipeline (mentioned in Appendix A, “Data Construction,” of the paper). We regard limited emotional coverage as a possible contributor, not an established cause. We will acknowledge the limited emotional range and the exploratory nature of this evaluation in the revised paper, and our ongoing work explores role-play scenarios with richer emotional context.

Table K. Exploratory evaluation of emotional expression

DiagnosticEx-OmniAudio2Face-3D
Happy-input clips classified as happy9/3022/30
Overall intended-emotion agreement37/179 (20.7%)49/179 (27.4%)

Response to Q2/W3(a)

We additionally test the commonly used FLAME2020 neutral template on 16 randomly selected clips, rendering the same audio and facial outputs on both Claire and FLAME2020. For this comparison, we align blendshape channels by name and use the same coefficient range [0,1] and 32 shared ARKit channels. FLAME2020 uses a transferred animation rig. The table reports mean Sync-D / Sync-C. The results show that Ex-Omni remains competitive.

Table L. Ablation study on different template

MethodClaire Sync-D ↓ / Sync-C ↑FLAME2020 Sync-D ↓ / Sync-C ↑
Ex-Omni8.679 / 5.97910.139 / 4.995
Ex-Omni + Audio2Face-3D8.614 / 6.10311.050 / 4.706
Ex-Omni+D311.020 / 3.40310.349 / 4.960
Ex-Omni+D610.118 / 4.2089.870 / 5.581
Ex-Omni+EmoTalk9.608 / 4.87910.958 / 4.802

Response to Q3/W3(b)

(1) For Table 12 in the paper appendix. Thank you for pointing this out. We have found that there is an error in the calculations presented in the table. The saved Audio2Face-3D outputs were at 60 fps, whereas the other methods were at 30 fps. We computed per-frame differences without first aligning the frame rates, which underestimated the teacher's motion statistics relative to the other methods. We have recalculated all methods at 30 fps and will correct the table and its interpretation in the revised paper. We also checked other results and didn't find the same problem. After correcting the frame rates, Ex-Omni’s acceleration is comparable to that of Audio2Face-3D (Acc-L2: 0.132 vs. 0.131), rather than substantially higher as previously reported.

Table M. Velocity and acceleration evaluation for generated ARKit-52 blendshape sequences. Corrected version of Table 12 in the paper.

MethodVel-AbsVel-L2Acc-AbsAcc-L2
Ex-Omni+D36.834 × 10⁻³7.496 × 10⁻²5.849 × 10⁻³6.182 × 10⁻²
Ex-Omni+D66.403 × 10⁻³7.812 × 10⁻²5.402 × 10⁻³6.292 × 10⁻²
Ex-Omni+EmoTalk1.251 × 10⁻²1.413 × 10⁻¹1.017 × 10⁻²1.109 × 10⁻¹
Ex-Omni + Audio2Face-3D9.963 × 10⁻³1.545 × 10⁻¹8.752 × 10⁻³1.312 × 10⁻¹
Ex-Omni9.687 × 10⁻³1.390 × 10⁻¹9.532 × 10⁻³1.317 × 10⁻¹

(2) Our evidence for perceptual quality comes from the human evaluation in Table 4 in the paper, which jointly assesses synchronization and naturalness. Ex-Omni receives 41.8% of preferences, compared with 1.9%, 0.3%, and 0.2% for the EmoTalk, D3, and D6 cascades, respectively; the Audio2Face-3D teacher receives 55.8%. This supports human preference for Ex-Omni over the non-teacher baselines.

(3) We additionally evaluate amplitude-normalized jerk and high-frequency power on CommonEval. Ex-Omni has higher normalized jerk than the teacher, but power above 6 Hz remains 1.458% for the mouth/jaw and 0.923% for the upper face, compared with 1.245% and 0.640% for the teacher. This indicates greater rapid temporal variation, rather than establishing perceptually visible jitter. The human evaluation in Table 4 in the paper provides complementary evidence: Ex-Omni receives 41.8% of preferences for joint synchronization and naturalness, substantially exceeding the non-teacher cascades, while the teacher remains preferred at 55.8%. The human study supports overall perceptual preference, but does not separately establish the absence of visible jitter.

Table N. Amplitude-normalized jerk and high-frequency power on CommonEval.

RegionMethodNormalized jerk ↓Power above 6 Hz (%)
Mouth/jawSaved reference4.504 × 10⁻¹1.245 × 10⁰
Mouth/jawEx-Omni5.285 × 10⁻¹1.458 × 10⁰
Mouth/jawEx-Omni + smoothing2.746 × 10⁻¹5.614 × 10⁻¹
Upper faceSaved reference3.226 × 10⁻¹6.405 × 10⁻¹
Upper faceEx-Omni4.186 × 10⁻¹9.229 × 10⁻¹
Upper faceEx-Omni + smoothing1.766 × 10⁻¹2.874 × 10⁻¹

Response to W3(c)

Following your suggestion, we additionally evaluate captured-motion prediction on VOCASET and BIWI Test-A, after adapting our pretrained model to their training splits. Tables C/D report vertex and lip-region errors, with an additional FDD metric on BIWI. LoRA improves adaptation over full-parameter fine-tuning, although Ex-Omni does not reach the strongest task-specific results. We observed overfitting during adaptation.We think that limited training-data scale and diversity may result this. For open-ended dialogue, independent SyncNet evaluation (mentioned in Sec. 5.1 of the paper) and the human study in Table 4 in the paper provide complementary evidence of synchronization and overall perceptual quality.

Response to Q4(a)

(1) We mentioned the Direct LLM-to-Face results in Table 11 of the paper. The results come from an extended training run. We report the best result from that run. Even with longer training, the variant did not converge well and achieved Sync-D / Sync-C of 16.395 / 0.292. This supports the optimization benefit of speech units under our tested training setup.

(2) We trained a stop-gradient variant that blocks face-loss gradients through the unit-generator hidden states while retaining speech supervision and face-decoder training. As shown in Table O, allowing face-loss gradients to propagate back into the unit generator improves Sync-D and Sync-C in all four settings. These results support the benefit of facial co-supervision for learning speech-unit-generator representations that improve speech–face synchronization.

Table O. Ablation of facial-loss gradient propagation.

DatasetModeConfigurationSync-D ↓Sync-C ↑
AlpacaEvalSpeech2Facew/ face-gradient9.2335.385
AlpacaEvalSpeech2Facew/o face-gradient9.8154.988
CommonEvalSpeech2Facew/ face-gradient9.2125.363
CommonEvalSpeech2Facew/o face-gradient9.8064.900
AlpacaEvalText2Facew/ face-gradient9.3135.236
AlpacaEvalText2Facew/o face-gradient9.8184.972
CommonEvalText2Facew/ face-gradient9.2395.403
CommonEvalText2Facew/o face-gradient9.8194.929
Reviewer_MXcoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Ex-Omni, a framework that enables omni-modal LLMs to generate text, speech, and synchronized 3D facial animation. The method uses speech units as intermediate representations and introduces token-as-query gated fusion (TQGF) to combine speech and semantic features. The authors also construct a 1.2M-sample dataset using synthetic speech and teacher-generated facial-motion labels. The experiments show improved synchronization compared with the evaluated non-teacher cascaded methods.

Critical strengths
  • The paper addresses an important problem by extending omni-modal LLMs to generate speech-synchronized 3D facial animation.
  • The use of speech units helps bridge the temporal gap between LLM representations and frame-level facial motion. The direct LLM-to-face ablation supports this design choice.
  • Ex-Omni achieves better synchronization than the evaluated non-teacher cascaded methods. Table 3 uses the same generated speech across different facial-animation methods, allowing a more controlled comparison.
  • The paper includes synchronization metrics, speech QA evaluation, human preference results, and ablation studies. These experiments help assess different aspects of the proposed system.
Critical weaknesses
  • The novelty of TQGF is unclear. The module combines cross-attention, residual connections, and gating, which are already widely used. The paper does not clearly distinguish TQGF from existing fusion mechanisms.
  • The ablation study does not fully isolate the contribution of TQGF. Table 7 compares the proposed module with single-source conditioning and feature concatenation, but does not report quantitative results for standard cross-attention without gating. This comparison is needed to determine whether the gating mechanism provides an improvement.
  • The method relies heavily on teacher-generated facial-motion labels. The training labels come from Audio2Face-3D, so the performance approaching the teacher may mainly reflect successful knowledge distillation. Evaluation on real facial-motion data would help determine whether the model generalizes beyond the teacher's outputs.
  • The evaluation focuses mainly on lip synchronization. SyncNet metrics measure audio-visual synchronization but do not directly assess facial expressiveness or naturalness. Additional evaluation of emotional expressions and upper-face movements is needed to support the claims about expressive facial animation.
  • Generalization across speakers is not sufficiently evaluated. The main synthetic training subset uses a single speaker identity. It remains unclear how the model performs with unseen speakers, different accents, speaking rates, and emotional speech.
  • The efficiency comparison is incomplete. Table 9 reports component-level latency and throughput, but does not provide a matched end-to-end comparison with the Audio2Face-3D cascade. The paper also lacks an analysis of latency and memory consumption for longer speech responses.
Questions for authors
  • How does TQGF compare with standard cross-attention without gating under identical training settings and model capacity?
  • How well does the model generalize to unseen speakers and different speaking styles?
  • What are the end-to-end latency and memory requirements compared with the Audio2Face-3D cascade, particularly for longer responses?
4 comments
AuthorsOpenReview ↗

Response to Reviewer MXco

We appreciate the your detailed feedback on the method and experiments. We have added further analyses and address the concerns below.

Response to W1 & W2 & Q1

Cross-attention and gating are established operations. Our task-specific design separates timing from context: response-token embeddings query LLM states for speech generation; frame-resampled speech-unit embeddings query unit-generator states for facial generation (Eqs. 8–10). We will emphasize this organization, not novelty of gating itself. To address your concern, we use the standard cross-attention to replace TQGF. As shown in Table E, Ex-Omni with TQGF achieves better Sync-D and Sync-C than with the standard cross-attention in all four settings.

Table E. Ablation study on TQGF.

DatasetmodeModuleSync-D ↓Sync-C ↑
AlpacaEvalSpeech2FaceTQGF9.2335.385
AlpacaEvalSpeech2Facestandard cross-attention9.8605.011
CommonEvalSpeech2FaceTQGF9.2125.363
CommonEvalSpeech2Facestandard cross-attention9.8195.112
AlpacaEvalText2FaceTQGF9.3135.236
AlpacaEvalText2Facestandard cross-attention9.8535.185
CommonEvalText2FaceTQGF9.2395.403
CommonEvalText2Facestandard cross-attention9.7825.072

Response to W3

(1) Without dataset-specific fine-tuning, we evaluate Ex-Omni and Audio2Face-3D on the complete Test-B audio splits of VOCASET and BIWI. As shown in Table F, Ex-Omni achieves Sync-D / Sync-C of 8.438 / 5.205 on VOCASET, compared with 7.917 / 5.866 for the teacher. On BIWI, the corresponding results are 8.543 / 4.519 for Ex-Omni and 8.058 / 5.058 for the teacher. Both retain audio-visual synchronization on these real recordings. This supports transfer to real speech inputs, rather than agreement with captured facial motion.

(2) Our advantage is native, coordinated text, speech, and 3D facial generation within an omni-modal dialogue model. Ex-Omni connects language representations with frame-level facial motion through speech-unit temporal structure and cross-modal fusion, while retaining speech-QA capability. At inference, its native facial branch generates animation without invoking the external Audio2Face-3D teacher. This extends the dialogue model's output capabilities even though a specialized teacher retains higher standalone synchronization scores. Because facial supervision comes from Audio2Face-3D pseudo-labels, Ex-Omni is encouraged to reproduce its motion patterns and may inherit its biases and restricted expressive diversity (mentioned in Appendix A and the Limitations section of the paper). The zero-shot synchronization results above do not demonstrate facial behavior beyond the teacher's distribution. Our 1.2M-sample weakly supervised dataset makes this integration trainable at scale, but cannot replace diverse captured conversational motion. We will explicitly discuss this teacher dependence in the Limitations section of the revised paper.

Table F. Zero-shot evaluation on the VOCASET and BIWI.

DatasetMethodSync-D ↓Sync-C ↑
VOCASETEx-Omni8.4385.205
VOCASETAudio2Face-3D7.9175.866
BIWIEx-Omni8.5434.519
BIWIAudio2Face-3D8.0585.058

Response to W4

(1) We use “expressive facial animation” broadly to include natural and coherent facial behavior, and speech–face coordination. Our goal is also to give an omni-modal LLM a human-like visual presence: an animated face that accompanies its spoken responses, extending interaction beyond text and voice. Thus, our contribution mainly encompasses natural avatar-based communication. The human study in Table 4 in the paper explicitly evaluates synchronization and naturalness jointly (mentioned in Appendix A, “Human Evaluation Protocol,” of the paper). These results support overall perceptual quality relative to the non-teacher cascades, while the teacher remains stronger. The study does not independently assess emotional expression or upper-face naturalness.

(2) For qualitative evidence of natural facial behavior, Figure 2, Case 2 in the paper illustrates natural mouth closure after speech ends. Additionally, we evaluate Ex-Omni on 179 CREMA-D recordings using HSEmotion. As shown in Table G, HSEmotion classifies 9/30 happy-input outputs as happy, while overall agreement with the intended emotion is only 37/179 (20.7%); predictions are predominantly neutral or happy. This is exploratory automatic recognition on rendered faces, not validated human judgment. These classifier-based results suggest some smile-like expression but do not establish reliable emotion matching or broad emotional diversity.

(3) Our data primarily cover common conversational exchanges rather than emotionally diverse role-play. Most QA-style responses are close to neutral under our data-construction pipeline (mentioned in Appendix A, “Data Construction,” of the paper). We regard limited emotional coverage as a possible contributor to the observed results, not an established cause. Our ongoing work explores role-play scenarios, where explicit character, situation, and emotional context make emotion-specific evaluation particularly informative. We will acknowledge the limited emotional range and the exploratory nature of this evaluation in the revised paper's Limitations section.

Table G. Exploratory evaluation of emotional expression

DiagnosticEx-OmniAudio2Face-3D
Happy-input clips classified as happy9/3022/30
Overall intended-emotion agreement37/179 (20.7%)49/179 (27.4%)

Response to Q2/W5

We distinguish robustness to different speech inputs from diversity of the model's generated voice.

(1) We supplied speech with different voice timbres and speaking rates, and Ex-Omni continued to generate facial-animation outputs across all tested conditions without voice-specific adaptation. Specifically, we synthesized 100 utterances from 10 newly designed voices using Qwen3-TTS-VoiceDesign and another 100 from 10 cloned voices, using reference recordings from a sampled subset of the VOCASET/BIWI test splits. Each voice reads the same 10 English sentences. At normal speed, Ex-Omni achieves Sync-D / Sync-C of 8.909 / 5.355 on designed voices and 8.555 / 5.639 on cloned voices.

Table H. Facial-generation robustness to different input voices and speaking rates

Audio sourceRateClips / voice conditionsEx-Omni (Sync-D ↓ / Sync-C ↑)AUdio2Face-3D (Sync-D ↓ / Sync-C ↑)
Original VOCASET/BIWI (sampled subset)0.5×20 / 107.970 / 3.2867.528 / 4.078
Original VOCASET/BIWI (sampled subset)0.8×20 / 108.341 / 4.4837.803 / 5.130
Original VOCASET/BIWI (sampled subset)1.0×20 / 108.652 / 4.6048.065 / 5.260
Original VOCASET/BIWI (sampled subset)1.2×20 / 108.953 / 4.4908.269 / 5.251
Original VOCASET/BIWI (sampled subset)1.5×20 / 109.668 / 3.5818.444 / 5.171
Qwen VoiceDesign0.5×100 / 108.930 / 4.0428.236 / 4.973
Qwen VoiceDesign0.8×100 / 108.947 / 5.0728.402 / 5.787
Qwen VoiceDesign1.0×100 / 108.909 / 5.3558.489 / 5.915
Qwen VoiceDesign1.2×100 / 109.260 / 4.9668.665 / 5.675
Qwen VoiceDesign1.5×100 / 1010.292 / 3.6588.791 / 5.305
Qwen reference-voice cloning0.5×100 / 108.321 / 4.1697.898 / 4.793
Qwen reference-voice cloning0.8×100 / 108.483 / 5.3857.920 / 6.049
Qwen reference-voice cloning1.0×100 / 108.555 / 5.6398.158 / 6.148
Qwen reference-voice cloning1.2×100 / 108.959 / 5.2508.433 / 5.860
Qwen reference-voice cloning1.5×100 / 109.657 / 4.3018.674 / 5.403

(2) We additionally test 179 CREMA-D recordings from 10 actors, covering neutral, happy, angry, sad, fearful, and disgusted speech, without supplying emotion labels to Ex-Omni. An automatic facial-expression recognizer (HSEmotion) classifies 9/30 happy-input animations as happy, but overall agreement with the intended input emotion is only 37/179 (20.7%); predictions are predominantly neutral or happy. This is exploratory automatic recognition on rendered faces, not validated human judgment. These classifier-based results suggest some smile-like expression but do not establish reliable emotion matching or broad emotional diversity. More details can be found in Response to W4.

(3) Accented-speech understanding is evaluated in Table 5 in the paper through the SD-QA benchmark in VoiceBench. Ex-Omni achieves 50.71, ranking second among the compared models (Qwen2.5-Omni-7B: 55.71). This supports understanding accented spoken questions, rather than generation of accented output speech.

(4) The current Ex-Omni model does not consider diverse output-voice settings, and our reported experiments do not establish native generation with multiple speaker voices. In our ongoing role-play work, we have introduced reference speech as additional conditioning to explore more diverse output voices.

Response to Q3/W6

We have further optimized our implementation on vLLM-Omni since submission and added measurements of the time required to generate a complete audio-visual response using this optimized implementation. We measure the entire process from text input to the finished video, including text generation, speech generation, facial animation, rendering, and encoding. Ex-Omni and Ex-Omni + Audio2Face-3D use the same generated text and speech, the same rendering settings. As shown in Table I, Ex-Omni reduces median total generation time by 6.1%–14.3% relative to the Audio2Face-3D cascade and uses approximately 2.9 GiB less GPU memory on these longer responses. With these further optimizations, Ex-Omni's complete-response RTF is below 1.0 for all tested lengths in Table I, improving on the earlier implementation reported in the paper. We will update the earlier RTF statement in the paper's Limitations section to reflect these results.

Table I. Additional latency analysis

Output duration (s)MethodComplete-video latency (s) ↓RTF ↓Sampled peak GPU memory (GiB) ↓
12.237Ex-Omni7.4200.60623.037
12.237Ex-Omni + Audio2Face-3D8.6600.70825.938
28.874Ex-Omni15.7610.54623.502
28.874Ex-Omni + Audio2Face-3D17.2390.59726.404
53.836Ex-Omni30.5110.56724.338
53.836Ex-Omni + Audio2Face-3D32.5090.60427.160

Dear Reviewer MXco,

Thank you for your detailed feedback and recognition of our speech-unit temporal scaffold. Our responses include the following additions:

  • Fusion Ablation: we compared TQGF with standard cross-attention, showing improvements in both synchronization metrics across all four settings (Response to W1/W2/Q1).
  • Generalization: added zero-shot tests on real recordings, voice and speaking-rate robustness tests, and exploratory emotion diagnostics (Response to W3–W5/Q2).
  • Efficiency: provided matched end-to-end latency and memory comparisons with the teacher cascade, including longer responses (Response to W6/Q3).

We also clarified the scope of TQGF’s contribution, teacher dependence, and limitations in emotional and output-voice diversity.

We would be grateful if you could take another look at these responses and share whether they address your main concerns. We would be happy to clarify any remaining questions.

Best regards,

The Authors

Reviewer_MijpOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores an interesting problem which transforms an omini-model large language model for joint generation of both speech and 3D facial animation. Different from conventional audo to 3D facial animation methods, it mainly focuses on generation response 3D facial motions from input texts. Meanwhile, it utilize a cascaded pipeline to automatically generate a very large scale dataset containing diverse audios and motions.

Critical strengths

(1) Taming a large language model for 3D generation (3D human motion or 3D facial motion) is a very promising and reasonable direction. I recognize the novelty of this paper.

Critical weaknesses

(1) Despite the novelty of this paper, the quality of the proposed dataset remains a problem. Directly using a pretrained audio to 3D faciaol motion generation method (Audio2Face-3D) to generate a dataset seems to be not very reliable. I am wondering if the proposed dataset will be public and I need some visualization results of the dataset to validate the quality of the proposed dataset. Otherwise, I think utilizing a 3D face reconstruction method to construct a dataset from in-the-wild videos will be a more realiable way for scaling up training data.

(2) The evaluation metrics seem to be limited. As audio to facial animation mainly utilize lip vertices error and FDD for evaluation. I think these two metrics can be added.

(3) Although this paper focuses on synthesizing response speed and 3D faces. Considering that the propose method is quite simple, I think the performance on directly generating speech and 3D motions from texts should be reported on VOCASET and BIWI, compared with "Avatron: 3D Face Avatar and Voice Generation From Text".

Questions for authors

See Weaknesses. If the authors can answer well, I will consider raise the score.

10 comments
AuthorsOpenReview ↗

Response to Reviewer Mijp

Thank you for recognizing the novelty of our work and for the suggestions on data quality and evaluation! We respond to each point below.

Response to W1

(1) We will release the complete dataset on Hugging Face upon acceptance. The anonymous repository provided at submission already contains the core training data in JSON format in the Constructed_Training_Data folder (stage1.json, stage2.json, and stage3.json).

(2) Our anonymous repository also includes a demo of the model trained on this dataset. Together with the visualizations in the paper, it shows that the model can learn from these pseudo-labels to generate speech-synchronized facial animation. This supports the dataset's practical value for training an omni-modal dialogue model with facial-animation capabilities.

(3) Additionally, we have uploaded 100 Stage2 speech–face pairs and 100 Stage3 Q&A pairs (including rendered videos) for your review. These samples are available in the data_visualization_for_review folder of our anonymous repository. Each includes the blendshape NPY, audio file, rendered video, and text.

(4) Although our method reduces the amount of data requirements, it still requires large-scale paired dialogue, speech, and consistently parameterized facial motion. Obtaining sufficiently diverse dialogue content with matching recordings and reliable facial reconstruction from in-the-wild videos is challenging. Synthetic construction lets us control dialogue coverage and transfer motion priors from a specialized model into a general multimodal model. We treat the resulting labels as weak supervision (mentioned in Sec. 4 and Appendix A of the paper) and acknowledge teacher bias.

Response to W2

We did not report these reference-based metrics because facial behavior in open-ended dialogue allows multiple valid realizations. Treating Audio2Face-3D generated pseudo-labels as ground truth could bias the comparison between methods. We therefore use SyncNet as an independent third-party evaluator (mentioned in Sec. 5.1 of the paper). Following your suggestion, we additionally report reference-based vertex and upper-face dynamics errors on CommonEval and AlpacaEval in Tables A/B, using Audio2Face-3D outputs as reference sequences. Ex-Omni achieves closer agreement with the teacher reference than the reported non-teacher baselines.

Table A. Additional performance comparison of Speech-to-Face.

DatasetMethodLmeanL_{mean} ↓ (u²)Squared LVE ↓ (u²)Mean absolute FDD ↓ (u²)
CommonEvalQwen2.5-Omni-3B+D31.258 × 10⁻²3.360 × 10⁻¹8.381 × 10⁻³
CommonEvalQwen2.5-Omni-3B+D69.502 × 10⁻³3.153 × 10⁻¹3.120 × 10⁻³
CommonEvalQwen2.5-Omni-3B+EmoTalk1.359 × 10⁻²4.223 × 10⁻¹1.627 × 10⁻²
CommonEvalQwen2.5-Omni-7B+D31.375 × 10⁻²3.727 × 10⁻¹8.325 × 10⁻³
CommonEvalQwen2.5-Omni-7B+D61.080 × 10⁻²3.581 × 10⁻¹3.406 × 10⁻³
CommonEvalQwen2.5-Omni-7B+EmoTalk1.534 × 10⁻²4.851 × 10⁻¹1.427 × 10⁻²
CommonEvalEx-Omni+D31.007 × 10⁻²2.859 × 10⁻¹6.157 × 10⁻³
CommonEvalEx-Omni+D67.548 × 10⁻³2.639 × 10⁻¹2.029 × 10⁻³
CommonEvalEx-Omni+EmoTalk1.014 × 10⁻²3.383 × 10⁻¹1.316 × 10⁻²
CommonEvalEx-Omni2.189 × 10⁻³7.950 × 10⁻²6.161 × 10⁻⁴
AlpacaEvalQwen2.5-Omni-3B+D31.259 × 10⁻²3.400 × 10⁻¹8.427 × 10⁻³
AlpacaEvalQwen2.5-Omni-3B+D69.696 × 10⁻³3.246 × 10⁻¹3.115 × 10⁻³
AlpacaEvalQwen2.5-Omni-3B+EmoTalk1.386 × 10⁻²4.382 × 10⁻¹1.625 × 10⁻²
AlpacaEvalQwen2.5-Omni-7B+D31.419 × 10⁻²3.952 × 10⁻¹8.577 × 10⁻³
AlpacaEvalQwen2.5-Omni-7B+D61.143 × 10⁻²3.852 × 10⁻¹3.746 × 10⁻³
AlpacaEvalQwen2.5-Omni-7B+EmoTalk1.641 × 10⁻²5.189 × 10⁻¹1.441 × 10⁻²
AlpacaEvalEx-Omni+D31.113 × 10⁻²3.134 × 10⁻¹6.458 × 10⁻³
AlpacaEvalEx-Omni+D68.517 × 10⁻³2.926 × 10⁻¹2.015 × 10⁻³
AlpacaEvalEx-Omni+EmoTalk1.095 × 10⁻²3.623 × 10⁻¹1.348 × 10⁻²
AlpacaEvalEx-Omni2.534 × 10⁻³9.262 × 10⁻²5.063 × 10⁻⁴

Table B. Additional performance comparison of Text-to-Face.

DatasetMethodLmeanL_{mean} ↓ (u²)Squared LVE ↓ (u²)Mean absolute FDD ↓ (u²)
CommonEvalQwen2.5-Omni-3B+D61.135 × 10⁻²3.597 × 10⁻¹1.345 × 10⁻³
CommonEvalQwen2.5-Omni-3B+EmoTalk1.383 × 10⁻²4.292 × 10⁻¹1.626 × 10⁻²
CommonEvalQwen2.5-Omni-7B+D31.344 × 10⁻²3.649 × 10⁻¹8.427 × 10⁻³
CommonEvalQwen2.5-Omni-7B+D61.061 × 10⁻²3.533 × 10⁻¹3.538 × 10⁻³
CommonEvalQwen2.5-Omni-7B+EmoTalk1.511 × 10⁻²4.789 × 10⁻¹1.448 × 10⁻²
CommonEvalEx-Omni+D31.002 × 10⁻²2.828 × 10⁻¹6.397 × 10⁻³
CommonEvalEx-Omni+D67.492 × 10⁻³2.617 × 10⁻¹2.045 × 10⁻³
CommonEvalEx-Omni+EmoTalk1.034 × 10⁻²3.422 × 10⁻¹1.311 × 10⁻²
CommonEvalEx-Omni2.111 × 10⁻³7.718 × 10⁻²5.261 × 10⁻⁴
AlpacaEvalQwen2.5-Omni-3B+D31.276 × 10⁻²3.422 × 10⁻¹8.606 × 10⁻³
AlpacaEvalQwen2.5-Omni-3B+D69.758 × 10⁻³3.236 × 10⁻¹3.298 × 10⁻³
AlpacaEvalQwen2.5-Omni-3B+EmoTalk1.373 × 10⁻²4.309 × 10⁻¹1.624 × 10⁻²
AlpacaEvalQwen2.5-Omni-7B+D31.418 × 10⁻²3.881 × 10⁻¹8.505 × 10⁻³
AlpacaEvalQwen2.5-Omni-7B+D61.138 × 10⁻²3.774 × 10⁻¹3.797 × 10⁻³
AlpacaEvalQwen2.5-Omni-7B+EmoTalk1.576 × 10⁻²5.014 × 10⁻¹1.438 × 10⁻²
AlpacaEvalEx-Omni+D31.112 × 10⁻²3.112 × 10⁻¹6.381 × 10⁻³
AlpacaEvalEx-Omni+D68.453 × 10⁻³2.900 × 10⁻¹1.985 × 10⁻³
AlpacaEvalEx-Omni+EmoTalk1.117 × 10⁻²3.657 × 10⁻¹1.400 × 10⁻²
AlpacaEvalEx-Omni2.634 × 10⁻³9.623 × 10⁻²5.042 × 10⁻⁴

Response to W3

Thanks for your suggestion. We will discuss Avatron [1] in the paper. Ex-Omni predicts ARKit blendshape coefficients rather than mesh vertices. We explored two adaptation routes: fitting ARKit coefficients to the captured vertices, and replacing the blendshape decoder's output layer with a dataset-specific vertex-offset head. As shown in the Table C and Table D below, We reported the better result among the two ways. These results demonstrate that our dialogue model can adapt to captured-motion prediction, although its reported errors remain higher than those of task-specific models under the respective protocols. We consider limited data scale and diversity a plausible contributor: both datasets offer only hundreds of training samples. Validation improvements slow markedly after roughly 10–15 epochs, and prolonged training eventually increases validation error despite decreasing training loss. LoRA improves results but still not SOTA. These observations also demonstrate that large-scale data for training is necessary for LLM. However, collecting large and diverse paired conversational speech and facial motion in the wild is challenging. Using synthetic data is a reasonable choice.

Table C. Vertex errors on VOCASET Test-A. The results of other methods come from Avatron [1].

Method / sourceLmeanL_{mean} ↓ (×10⁻⁷)LmaxlipL^{lip}_{max} ↓ (×10⁻⁵)LmeanlipL^{lip}_{mean} ↓ (×10⁻⁶)
FaceFormer3.1521.4961.704
CodeTalker3.3871.7091.847
FaceDiffuser3.2691.5901.697
ScanTalk3.3581.8661.899
Avatron2.9771.3341.492
Avatron w/o phoneme-label embedding3.0321.4071.568
Ex-Omni — full parameter fine-tuning3.9982.2002.551
Ex-Omni — LoRA3.3491.8161.959

Table D. Vertex errors on BIWI Test-A. The results of other methods come from Avatron [1].

Method / sourceLmeanL_{mean} ↓ (×10⁻⁵)LmaxlipL^{lip}_{max} ↓ (×10⁻⁴)LmeanlipL^{lip}_{mean} ↓ (×10⁻⁵)FDD³ (×10⁻⁵)
FaceFormer2.0564.2864.3182.078
CodeTalker2.5895.4495.3222.841
FaceDiffuser1.9283.8493.9552.023
Avatron1.9103.9403.9312.340
Avatron w/o phoneme-label embedding — Table I1.9904.1544.1422.220
Ex-Omni — full parameter fine-tuning3.9097.3147.4626.740
Ex-Omni — LoRA3.7437.0767.0866.026

Reference

[1] S. Um, Y. Lee, W. Ko, Y. Zhou, S. Lee and H. -G. Kang, "Avatron: 3D Face Avatar and Voice Generation From Text," in IEEE Transactions on Multimedia, vol. 28, pp. 4759-4770, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.