- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper extends an omni-modal LLM to generate speech-synchronized 3D facial animation (ARKit-52 blendshapes) along with text and speech. Ex-Omni is built on Qwen3-8B with a Whisper encoder. A Qwen3-0.6B speech-unit generator, in the style of GLM-4-Voice, produces discrete speech units conditioned on response tokens and LLM hidden states. A non-autoregressive blendshape decoder uses the unit embeddings, interpolated to frame rate, as queries and the unit generator's hidden states as context. Both connections use a gated cross-attention block (TQGF). Training data (InstructS2SF-1200K) is fully synthetic:
- text from Emilia and InstructS2S is voiced by Qwen3-TTS with a single speaker;
- NVIDIA Audio2Face-3D, conditioned on Audio2Emotion, produces blendshape pseudo-labels.
Ex-Omni beats OLLM + EmoTalk/UniTalker cascades on SyncNet, comes second to the Audio2Face-3D teacher cascade in SyncNet and human preference (41.8% vs. 55.8%), and keeps most of its speech-QA ability
Critical strengths
- Adding facial animation as an output of a speech-capable LLM is a practically relevant direction for avatars and embodied agents. Using speech units as a temporal scaffold for the face decoder is a sensible design.
- it is great that they explain why LVE against teacher outputs would be circular, add a controlled comparison with fixed upstream speech (Table 3), and openly include the teacher cascade as a stronger reference.
- They checked multiple evaluations including VoiceBench QA, UTMOS, speech–text WER, and text-only retention against the base Qwen3-8B. Also they report latency.
- The code, data pipeline, and 20 randomly sampled supplementary videos aid reproducibility and qualitative judgment.
Critical weaknesses
-
The native face branch is a distilled student that doesn't beat its teacher, and the benefit of going native is not shown. In every face metric the teacher cascade (Ex-Omni speech → Audio2Face-3D) is better: Sync-D 8.92 vs. 9.21 and human preference 55.8% vs. 41.8% (Tables 3–4). So the practical alternative, running an OLLM and piping its speech into Audio2Face-3D, gives better faces today. The paper's argument for integration is avoiding an external S2F model at inference, but it never reports the latency or compute of the teacher cascade. The limitations section also says the overall real-time factor is above 1.0 and the latency gain is limited to the face branch. To justify the "native" framing, the paper needs to show some advantage the cascade cannot have. Examples: lower end-to-end latency or first-frame time, streaming, or facial behavior that depends on dialogue semantics rather than audio alone. Otherwise the contribution reduces to distilling Audio2Face-3D into an OLLM head.
-
"Expressive" is not demonstrated. The introduction motivates the work with non-verbal cues in face-to-face interaction. But Sec. 1 redefines "expressive" as articulation and lip-sync. Appendix A notes that most pseudo-labels in the dialogue data are close to neutral, and the facial labels are a function of the audio only. The model therefore has no training signal linking dialogue content or intent to facial expression, which is what an LLM-integrated face model could uniquely offer. Nothing is evaluated for emotion, expression diversity, or alignment between text semantics and face. The name and motivation promise more than the method delivers.
-
The evaluation is weak for 3D facial animation.
- SyncNet is trained on real 2D talking faces and is applied here to renders of one ARKit template (Claire, the Audio2Face template). EmoTalk and UniTalker blendshape outputs may follow different conventions and scales than Audio2Face-3D's ARKit rig. Rendering everything on the teacher's native template may favor the teacher and its student independently of motion quality.
- The temporal diagnostics (Table 12) show Ex-Omni's acceleration is about 3× the teacher's (Acc-L2 0.132 vs. 0.043). The paper describes this as "greater frame-to-frame variation," but high acceleration with a plain frame-wise MSE loss and no smoothness term usually means jitter. This needs a direct check, such as a jitter metric, a human rating of smoothness, or the supplementary videos at full frame rate.
- No evaluation uses real captured facial motion. One option: tokenize real speech from a mocap set (e.g., the 3D-ETF/RAVDESS-based sets used by EmoTalk) into units and compare against ground truth. That would show whether the student learned facial motion or Audio2Face-3D's specific style.
Questions for authors
- Can you show any facial behavior that depends on dialogue content beyond the audio, which a speech-to-face cascade could not produce?
- Were all baselines' blendshape outputs mapped onto the same ARKit convention and scale before rendering? How do SyncNet results change on a neutral template not used by Audio2Face-3D?
- Is the higher acceleration in Table 12 jitter? Could you report a jitter or smoothness metric, or a human smoothness rating?
- How does the Direct LLM-to-Face variant perform when trained to convergence? What happens if you stop gradients from the face decoder into the unit generator?
6 comments
Response to Q3/W3(b)
(1) For Table 12 in the paper appendix. Thank you for pointing this out. We have found that there is an error in the calculations presented in the table. The saved Audio2Face-3D outputs were at 60 fps, whereas the other methods were at 30 fps. We computed per-frame differences without first aligning the frame rates, which underestimated the teacher's motion statistics relative to the other methods. We have recalculated all methods at 30 fps and will correct the table and its interpretation in the revised paper. We also checked other results and didn't find the same problem. After correcting the frame rates, Ex-Omni’s acceleration is comparable to that of Audio2Face-3D (Acc-L2: 0.132 vs. 0.131), rather than substantially higher as previously reported.
Table M. Velocity and acceleration evaluation for generated ARKit-52 blendshape sequences. Corrected version of Table 12 in the paper.
| Method | Vel-Abs | Vel-L2 | Acc-Abs | Acc-L2 |
|---|---|---|---|---|
| Ex-Omni+D3 | 6.834 × 10⁻³ | 7.496 × 10⁻² | 5.849 × 10⁻³ | 6.182 × 10⁻² |
| Ex-Omni+D6 | 6.403 × 10⁻³ | 7.812 × 10⁻² | 5.402 × 10⁻³ | 6.292 × 10⁻² |
| Ex-Omni+EmoTalk | 1.251 × 10⁻² | 1.413 × 10⁻¹ | 1.017 × 10⁻² | 1.109 × 10⁻¹ |
| Ex-Omni + Audio2Face-3D | 9.963 × 10⁻³ | 1.545 × 10⁻¹ | 8.752 × 10⁻³ | 1.312 × 10⁻¹ |
| Ex-Omni | 9.687 × 10⁻³ | 1.390 × 10⁻¹ | 9.532 × 10⁻³ | 1.317 × 10⁻¹ |
(2) Our evidence for perceptual quality comes from the human evaluation in Table 4 in the paper, which jointly assesses synchronization and naturalness. Ex-Omni receives 41.8% of preferences, compared with 1.9%, 0.3%, and 0.2% for the EmoTalk, D3, and D6 cascades, respectively; the Audio2Face-3D teacher receives 55.8%. This supports human preference for Ex-Omni over the non-teacher baselines.
(3) We additionally evaluate amplitude-normalized jerk and high-frequency power on CommonEval. Ex-Omni has higher normalized jerk than the teacher, but power above 6 Hz remains 1.458% for the mouth/jaw and 0.923% for the upper face, compared with 1.245% and 0.640% for the teacher. This indicates greater rapid temporal variation, rather than establishing perceptually visible jitter. The human evaluation in Table 4 in the paper provides complementary evidence: Ex-Omni receives 41.8% of preferences for joint synchronization and naturalness, substantially exceeding the non-teacher cascades, while the teacher remains preferred at 55.8%. The human study supports overall perceptual preference, but does not separately establish the absence of visible jitter.
Table N. Amplitude-normalized jerk and high-frequency power on CommonEval.
| Region | Method | Normalized jerk ↓ | Power above 6 Hz (%) |
|---|---|---|---|
| Mouth/jaw | Saved reference | 4.504 × 10⁻¹ | 1.245 × 10⁰ |
| Mouth/jaw | Ex-Omni | 5.285 × 10⁻¹ | 1.458 × 10⁰ |
| Mouth/jaw | Ex-Omni + smoothing | 2.746 × 10⁻¹ | 5.614 × 10⁻¹ |
| Upper face | Saved reference | 3.226 × 10⁻¹ | 6.405 × 10⁻¹ |
| Upper face | Ex-Omni | 4.186 × 10⁻¹ | 9.229 × 10⁻¹ |
| Upper face | Ex-Omni + smoothing | 1.766 × 10⁻¹ | 2.874 × 10⁻¹ |
Response to W3(c)
Following your suggestion, we additionally evaluate captured-motion prediction on VOCASET and BIWI Test-A, after adapting our pretrained model to their training splits. Tables C/D report vertex and lip-region errors, with an additional FDD metric on BIWI. LoRA improves adaptation over full-parameter fine-tuning, although Ex-Omni does not reach the strongest task-specific results. We observed overfitting during adaptation.We think that limited training-data scale and diversity may result this. For open-ended dialogue, independent SyncNet evaluation (mentioned in Sec. 5.1 of the paper) and the human study in Table 4 in the paper provide complementary evidence of synchronization and overall perceptual quality.
Response to Q4(a)
(1) We mentioned the Direct LLM-to-Face results in Table 11 of the paper. The results come from an extended training run. We report the best result from that run. Even with longer training, the variant did not converge well and achieved Sync-D / Sync-C of 16.395 / 0.292. This supports the optimization benefit of speech units under our tested training setup.
(2) We trained a stop-gradient variant that blocks face-loss gradients through the unit-generator hidden states while retaining speech supervision and face-decoder training. As shown in Table O, allowing face-loss gradients to propagate back into the unit generator improves Sync-D and Sync-C in all four settings. These results support the benefit of facial co-supervision for learning speech-unit-generator representations that improve speech–face synchronization.
Table O. Ablation of facial-loss gradient propagation.
| Dataset | Mode | Configuration | Sync-D ↓ | Sync-C ↑ |
|---|---|---|---|---|
| AlpacaEval | Speech2Face | w/ face-gradient | 9.233 | 5.385 |
| AlpacaEval | Speech2Face | w/o face-gradient | 9.815 | 4.988 |
| CommonEval | Speech2Face | w/ face-gradient | 9.212 | 5.363 |
| CommonEval | Speech2Face | w/o face-gradient | 9.806 | 4.900 |
| AlpacaEval | Text2Face | w/ face-gradient | 9.313 | 5.236 |
| AlpacaEval | Text2Face | w/o face-gradient | 9.818 | 4.972 |
| CommonEval | Text2Face | w/ face-gradient | 9.239 | 5.403 |
| CommonEval | Text2Face | w/o face-gradient | 9.819 | 4.929 |
Response to Reviewer enYo
We really thank you for the careful evaluation and constructive suggestions! Below, we clarify the benefits of native generation and present additional evaluations.
Response to W1
The practical benefit of native generation is that one dialogue model generates text, speech, and facial animation without deploying separate Audio2Emotion and Audio2Face-3D models. Our main contribution is to give a multimodal dialogue model this capability within its own generation process. Since submission, we have further optimized our implementation on vLLM-Omni and measured the time from text input to a finished audio-visual response using this optimized implementation. As shown in Table J, Ex-Omni reduces median total generation time by 3.0%–14.3% relative to the Audio2Face-3D cascade across the tested response lengths. Complete-response generation takes less time than playback for all tested lengths, improving on the earlier implementation. We will update the earlier RTF statement in the paper's Limitations section to reflect these results.
Table J. Additional latency analysis
Response to Q1/W2
(1) We keep the response text, speech units, and audio unchanged, and change only the dialogue context. In some comparisons, facial coefficients change in the expected direction. However, the changes are small. This suggests that dialogue context can affect facial generation.
(2) We include natural facial behavior and speech–face coordination, as well as emotional expression. Our aim is to give an omni-modal dialogue model a human-like visual presence alongside its text and speech. The human study in Table 4 in the paper evaluates synchronization and naturalness jointly (mentioned in Appendix A, “Human Evaluation Protocol,” of the paper) and supports overall perceptual quality relative to the non-teacher cascades; the specialized teacher remains stronger. The study does not independently assess emotional expression or upper-face naturalness. Figure 2, Case 2 in the paper illustrates natural mouth closure after speech ends.
(3) In our additional CREMA-D evaluation, as shown in Table K, HSEmotion classifies 9/30 happy-input outputs as happy, while overall agreement with the intended emotion is only 37/179 (20.7%); predictions are predominantly neutral or happy. This is exploratory automatic recognition on rendered faces, not validated human judgment. These classifier-based results suggest some smile-like expression but do not establish reliable emotion matching or broad emotional diversity. Our training data focus on everyday conversations rather than emotionally diverse role-play, and most QA-style responses are close to neutral under our data-construction pipeline (mentioned in Appendix A, “Data Construction,” of the paper). We regard limited emotional coverage as a possible contributor, not an established cause. We will acknowledge the limited emotional range and the exploratory nature of this evaluation in the revised paper, and our ongoing work explores role-play scenarios with richer emotional context.
Table K. Exploratory evaluation of emotional expression
Response to Q2/W3(a)
We additionally test the commonly used FLAME2020 neutral template on 16 randomly selected clips, rendering the same audio and facial outputs on both Claire and FLAME2020. For this comparison, we align blendshape channels by name and use the same coefficient range [0,1] and 32 shared ARKit channels. FLAME2020 uses a transferred animation rig. The table reports mean Sync-D / Sync-C. The results show that Ex-Omni remains competitive.
Table L. Ablation study on different template