acceptodds
Withdrawn from ICLR 2027

ERTalk: Autoregressive Speech-Driven 3D Face Animation Guided by Expressive Reference Motion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Speech-driven 3D facial animation aims to generate natural and expressive facial motion with accurate lip synchronization. Discrete emotion labels alone provide limited information about fine-grained facial deformations. We introduce ERTalk, a reference-conditioned autoregressive model that uses speech and reference motion to generate synchronized lip movements and reference-guided expressions in a learned deformation space. Multi-State Reference Conditioning (MRC) augments the global reference prefix with layer-wise access to frame-level expression states, while differentiable decoding enables vertex and velocity supervision over all decoded vertices in valid frames. Our pretrained SMPL-X-based representation encodes facial expression separately from the target identity geometry. This representation allows the generated motion to be transferred across identities without rerunning the motion generator. Extensive evaluations demonstrate that our method outperforms existing approaches in both lip-sync accuracy and emotion control.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Digital Human/Autoregressive Generation/Speech-driven Animation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ckFMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes ERTalk, an autoregressive model that generates 3D facial animations from speech and a reference motion with a desired facial expression. The core module, Multi-State Reference Conditioning (MRC), integrates a global prefix that summarizes the entire reference with a frame-level reference memory to condition each layer throughout the generation process. Furthermore, after reconstructing 3D vertices from the predicted tokens, the model jointly applies vertex and velocity loss. The author demonstrates both qualitative and quantitative experiments that ERTalk achieves lower errors than baselines and higher preference in user studies.

Critical strengths
  • Validity of Problem Formulation: Discrete labels only assign emotion categories and fail to capture fine-grained expression variations (e.g., eyebrows, mouth corners, and intensity). This is a valid critique, and the direction of controlling emotion via a reference motion is highly compelling.

  • Performance in the Reported Evaluation: ERTalk outperforms five baselines across all quantitative metrics (MVE, LVE, EVE, and AFDD) with lower errors, while qualitative comparisons (Fig. 4) demonstrates that its lip opening and fine-grained emotion more closely match the ground truth (GT).

  • Geometry-aware Supervision: Leveraging differentiable decoding via Gumbel-Softmax, ERTalk directly applies vertex and velocity losses to discrete token prediction, consistently improving all four metrics in the ablation study (Table 3).

Critical weaknesses
  • Inappropriate and Missing Citations in the Discussion of Emotion Control: In L56, the authors introduce ARTalk as a source of emotional information; however, since ARTalk focuses on controlling speaking style, it does not fit the paragraph discussing fine-grained emotion control. Furthermore, the citation of EcoFace (Xie et al., ICLR 2025), which should be placed in this context, is missing.

  • Unsubstantiated Claim and Flawed Evaluation of Reference–Target Articulation Conflict: Although the authors argue that articulation in the reference motion can conflict with target audio pronunciation (L63–64), this claim remains completely unsubstantiated by either prior literature or empirical evidence. Architecturally, the model also lacks an explicit mechanism to disentangle articulatory movements from facial expressions within the reference motion. While the authors remove temporal positional encoding in the MRC memory (L213–214), this only disrupts temporal alignment without preventing the cross-attention layers from directly attending to reference lip shapes. Despite this, the paper provides no quantitative evaluation verifying whether target lip-sync accuracy is maintained under cross-utterance references, relying solely on a single qualitative case in Fig. 5. Above all, the training and evaluation protocols suffer from a critical flaw: training preferentially samples same-video references (L278), and both quantitative and ablation experiments exclusively use 20 consecutive reference frames following the target window in the same clip (L279–280). Because this setup represents a best-case scenario with minimal conflict and is infeasible in real-world applications, the reported metrics fail to reflect practical real-world performance.

  • Lack of validation on fine-grained emotion control: The authors claim fine-grained emotion control, but the experiments that control emotion by changing the reference motion (Fig. 1, 5, 6) cover only two emotions, happy and angry. The user study and Fig. 4 cover more emotions, but the paper does not explain why the control experiments show only two. In addition, Since the quantitative evaluation uses frames that follow the target window in the same clip as the reference, it is not clear whether the output follows the GT motion or the fine-grained emotional expression. The emotion accuracy in the user study is also judged based on the GT video (L405–407). Therefore, the paper does not show whether the output follows the emotion of a different reference motion. To confirm this, a quantitative metric that measures whether the output follows the emotion is needed, such as VE-FID from EcoFace.

  • Reliability of User study: Fifteen participants evaluated a total of 10 trials, with only two trials per baseline (L404–405). Additionally, the number of speakers in the samples used in the user study is not specified. In comparison with LSF-Animation, the preference for ERTalk in Lip/Face Acc. and Emotion Acc. is 56.7% [39.2, 72.6], and since the confidence interval includes 50%, the result is indistinguishable from the chance level. Therefore, the authors must provide statistical tests or other evidence that the results of the user study are reliable.

  • Unclear Source of MRC Performance Improvement: In Table 3, removing MRC greatly increases MVE from 0.528 to 0.933 and EVE from 0.476 to 1.034. Notably, since the MVE of w/o MRC is worse than that of ARTalk (0.855), most of the performance advantage over the baselines comes from MRC. However, it is unclear whether this improvement is the result of using emotion information from the reference or obtaining information similar to the GT from adjacent frames of the same clip. The fact that the LVE, the lip-region error, decreases by nearly 30% with MRC also suggests the latter. Therefore, experiments are needed to distinguish between the two possibilities, such as performing the same ablation with references from different clips or different utterances.

(Minor comments)

  1. Missing Citation: No citation for "spoken content alone does not determine how a face should express emotion" in L51–52. Missing citation for StreamingTalker (Yang et al., AAAI 2026), a recent work in this field. Missing citation for EmoFace (Liu et al., IEEE VR 2024), a work on emotion control.

  2. Lack of Evidence for the Importance of Emotion Control: The paper only claims that emotion, especially fine-grained emotion control, is important for realistic 3D facial animation(L048–053, L057–061), but gives no evidence for why.

Questions for authors
  1. Separating speech articulation (mouth shapes) from fine-grained emotion in facial motion is very ambiguous. How do the authors distinguish the two, and how can they show that the two are distinguished?
  2. In real use, does a user have to provide a 3D reference motion with fine-grained emotion to reflect that emotion? How can such a reference motion be created?
  3. Geometry-aware motion supervision, which the authors present as one of their contributions (decoding the predicted tokens into a mesh through differentiable decoding and applying vertex and velocity losses), shows almost no visible difference (Fig. 3(b)) and only a small numerical difference (Table 3). Are there any other samples that show a large difference?
  4. Where on the face do the authors believe emotions are reflected? The authors state that the same emotion can differ in eyebrow height, cheek deformation, and lip-corner motion (L057–059), and in the description of Fig. 4, they cite mouth asymmetry, lip corners, and mouth opening as emotional cues (L322–324). However, the emotion metrics EVE and AFDD measure only the upper face (L295–299), so only the eyebrows are covered among these cues. Emotional cues around the mouth are mixed with lip-sync errors in LVE and are not evaluated separately. Also, semantic regions (L292) are not defined, so it is unclear which metric includes cheek deformation. Why do the claimed regions differ from the evaluated regions, and how were emotional expressions around the mouth verified?
Reviewer_o4L5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ERTalk is speech driven 3D face motion generation model guided by a ref clip. It uses AR motion tokens and gated cross attn to frame level reference features.

Critical strengths
  • Reference motion can be more useful for providing fine expression control than a fixed emo category label.
  • The results and the user study seem to be promising.
Critical weaknesses
  • Quant eval uses 20 reference frames immediately after the target window in the same clip, which gives a closely matched reference. While different reference examples are only shown qualitatively in Fig. 5. Additionally, for the ARTALK retrained baseline, which uses original conditioning interfaces, its not clear if it uses the same reference as ERTalk for the quant evaluation?
  • The extra data contribution is not clear. Mainly, the authors use 205 hrs of data, but baselines have different training steps, so there's also no matching pretraining control.
  • ARTalk and Diffposetalk are similar to the proposed method, so the novelty is limited.
  • THe pretrain expression space and decoder pretraining details are not clear to me. While the authors describe the SMPLX expression representation, decoder, and losses, the pretrain data and full process for these components are not included.
  • For cross model comparison, the authors use FLAME - SMPLX vertex correspondence, but does this mapping affect the metrics in any way?
  • The paper claims identity transfer capabilities but includes no separate quantitative tests.
Questions for authors
  • Please explain how the proposed model performs with reference to another clip / emo / identity. Also explain the ARTALK input references.
  • Can the authors show the results without the 205 hr pretraining? And how was the pretraining expression space designed?
  • It's important to include DiffPosetalk in the baseline, as well as quantitative emo control and identity transfer experiments.
Reviewer_qwHhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ERTalk addresses the meaningful problem of reference-motion-guided expressive 3D facial animation and presents a relatively complete framework that combines autoregressive motion-token generation, multi-state reference conditioning, and geometry-aware supervision. The experimental results are reasonably convincing, particularly in terms of emotion control and lip synchronization.

Critical strengths

The proposed Multi-State Reference Conditioning simultaneously leverages a global reference condition and frame-level reference motion states, which helps capture local dynamic variations in the reference expression. The motivation behind this design is clear.

Critical weaknesses

Although Multi-State Reference Conditioning can leverage both global and frame-level reference motion information, it mainly relies on existing cross-attention and memory-conditioning mechanisms. Meanwhile, the independent contribution of the geometric supervision remains unclear. According to the ablation results, removing the geometry loss leads to only limited changes in the overall metrics. Therefore, the authors should further clarify whether the vertex and velocity losses provide indispensable performance gains or merely serve as conventional training aids. The paper should also compare MRC with simpler reference-conditioning strategies and conduct cross-identity and cross-video experiments to validate the necessity of MRC and its ability to disentangle reference expressions from the target speech.

Questions for authors

as my comment

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.