- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes ERTalk, an autoregressive model that generates 3D facial animations from speech and a reference motion with a desired facial expression. The core module, Multi-State Reference Conditioning (MRC), integrates a global prefix that summarizes the entire reference with a frame-level reference memory to condition each layer throughout the generation process. Furthermore, after reconstructing 3D vertices from the predicted tokens, the model jointly applies vertex and velocity loss. The author demonstrates both qualitative and quantitative experiments that ERTalk achieves lower errors than baselines and higher preference in user studies.
Critical strengths
-
Validity of Problem Formulation: Discrete labels only assign emotion categories and fail to capture fine-grained expression variations (e.g., eyebrows, mouth corners, and intensity). This is a valid critique, and the direction of controlling emotion via a reference motion is highly compelling.
-
Performance in the Reported Evaluation: ERTalk outperforms five baselines across all quantitative metrics (MVE, LVE, EVE, and AFDD) with lower errors, while qualitative comparisons (Fig. 4) demonstrates that its lip opening and fine-grained emotion more closely match the ground truth (GT).
-
Geometry-aware Supervision: Leveraging differentiable decoding via Gumbel-Softmax, ERTalk directly applies vertex and velocity losses to discrete token prediction, consistently improving all four metrics in the ablation study (Table 3).
Critical weaknesses
-
Inappropriate and Missing Citations in the Discussion of Emotion Control: In L56, the authors introduce ARTalk as a source of emotional information; however, since ARTalk focuses on controlling speaking style, it does not fit the paragraph discussing fine-grained emotion control. Furthermore, the citation of EcoFace (Xie et al., ICLR 2025), which should be placed in this context, is missing.
-
Unsubstantiated Claim and Flawed Evaluation of Reference–Target Articulation Conflict: Although the authors argue that articulation in the reference motion can conflict with target audio pronunciation (L63–64), this claim remains completely unsubstantiated by either prior literature or empirical evidence. Architecturally, the model also lacks an explicit mechanism to disentangle articulatory movements from facial expressions within the reference motion. While the authors remove temporal positional encoding in the MRC memory (L213–214), this only disrupts temporal alignment without preventing the cross-attention layers from directly attending to reference lip shapes. Despite this, the paper provides no quantitative evaluation verifying whether target lip-sync accuracy is maintained under cross-utterance references, relying solely on a single qualitative case in Fig. 5. Above all, the training and evaluation protocols suffer from a critical flaw: training preferentially samples same-video references (L278), and both quantitative and ablation experiments exclusively use 20 consecutive reference frames following the target window in the same clip (L279–280). Because this setup represents a best-case scenario with minimal conflict and is infeasible in real-world applications, the reported metrics fail to reflect practical real-world performance.
-
Lack of validation on fine-grained emotion control: The authors claim fine-grained emotion control, but the experiments that control emotion by changing the reference motion (Fig. 1, 5, 6) cover only two emotions, happy and angry. The user study and Fig. 4 cover more emotions, but the paper does not explain why the control experiments show only two. In addition, Since the quantitative evaluation uses frames that follow the target window in the same clip as the reference, it is not clear whether the output follows the GT motion or the fine-grained emotional expression. The emotion accuracy in the user study is also judged based on the GT video (L405–407). Therefore, the paper does not show whether the output follows the emotion of a different reference motion. To confirm this, a quantitative metric that measures whether the output follows the emotion is needed, such as VE-FID from EcoFace.
-
Reliability of User study: Fifteen participants evaluated a total of 10 trials, with only two trials per baseline (L404–405). Additionally, the number of speakers in the samples used in the user study is not specified. In comparison with LSF-Animation, the preference for ERTalk in Lip/Face Acc. and Emotion Acc. is 56.7% [39.2, 72.6], and since the confidence interval includes 50%, the result is indistinguishable from the chance level. Therefore, the authors must provide statistical tests or other evidence that the results of the user study are reliable.
-
Unclear Source of MRC Performance Improvement: In Table 3, removing MRC greatly increases MVE from 0.528 to 0.933 and EVE from 0.476 to 1.034. Notably, since the MVE of w/o MRC is worse than that of ARTalk (0.855), most of the performance advantage over the baselines comes from MRC. However, it is unclear whether this improvement is the result of using emotion information from the reference or obtaining information similar to the GT from adjacent frames of the same clip. The fact that the LVE, the lip-region error, decreases by nearly 30% with MRC also suggests the latter. Therefore, experiments are needed to distinguish between the two possibilities, such as performing the same ablation with references from different clips or different utterances.
(Minor comments)
-
Missing Citation: No citation for "spoken content alone does not determine how a face should express emotion" in L51–52. Missing citation for StreamingTalker (Yang et al., AAAI 2026), a recent work in this field. Missing citation for EmoFace (Liu et al., IEEE VR 2024), a work on emotion control.
-
Lack of Evidence for the Importance of Emotion Control: The paper only claims that emotion, especially fine-grained emotion control, is important for realistic 3D facial animation(L048–053, L057–061), but gives no evidence for why.
Questions for authors
- Separating speech articulation (mouth shapes) from fine-grained emotion in facial motion is very ambiguous. How do the authors distinguish the two, and how can they show that the two are distinguished?
- In real use, does a user have to provide a 3D reference motion with fine-grained emotion to reflect that emotion? How can such a reference motion be created?
- Geometry-aware motion supervision, which the authors present as one of their contributions (decoding the predicted tokens into a mesh through differentiable decoding and applying vertex and velocity losses), shows almost no visible difference (Fig. 3(b)) and only a small numerical difference (Table 3). Are there any other samples that show a large difference?
- Where on the face do the authors believe emotions are reflected? The authors state that the same emotion can differ in eyebrow height, cheek deformation, and lip-corner motion (L057–059), and in the description of Fig. 4, they cite mouth asymmetry, lip corners, and mouth opening as emotional cues (L322–324). However, the emotion metrics EVE and AFDD measure only the upper face (L295–299), so only the eyebrows are covered among these cues. Emotional cues around the mouth are mixed with lip-sync errors in LVE and are not evaluated separately. Also, semantic regions (L292) are not defined, so it is unclear which metric includes cheek deformation. Why do the claimed regions differ from the evaluated regions, and how were emotional expressions around the mouth verified?