- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a cross-lingual personalized speech-driven 3D facial animation model that preserves the speaker-specific motion style when driven by target-language speech, even without target-language motion. Previous studies have primarily used audio-derived style representations or reference motion to reflect the speaker's style. However, audio-derived representations have limitations in that the inferred style changes even for the same speaker when the language changes, as the speaker's style is entangled with language-dependent acoustic variation. Reference-based representations have limitations in that they are influenced by sample-specific factors such as emotion, speaking rate, and local expression. To overcome these limitations, the authors propose a method to learn a shared audio-motion style space and to regularize style representations extracted from source- and target-language speech based on source-language motion. Through various experiments, the authors demonstrate that the proposed method is effective in terms of motion accuracy, temporal consistency, and speaker-specific style preservation.
Critical strengths
-
Validity of the problem setting: The motivation that the speaker-specific motion style should be preserved even when the language changes is valid. Target-language speech can be collected relatively easily, whereas 3D facial motion requires specialized capture equipment and substantial post-processing (L041–043). Considering this, it is good that the authors propose a weakly supervised learning that uses only source-language speech-motion pairs and target-language speech.
-
Performance in bidirectional evaluation: In both English-to-non-English and non-English-to-English transfer directions, the proposed method shows the best performance across all seven metrics (LVE, FVE, Lip-max, LDTW, FDTW, FID, SCS) compared to five baselines (Table 2). The bidirectional evaluation shows that the performance of the proposed method does not depend on a specific source language.
-
Intuitive visualization through demo videos: In the demo videos in the supplementary materials, the results of GT, the proposed method, and five baselines can be compared side by side with error heatmaps, allowing for an intuitive view of the differences in results in both transfer directions.
Critical weaknesses
- Overlap with Mimic and Missing Comparison: The proposed method has many methodological overlaps with Mimic (Fu et al., AAAI 2024). Specifically, the following three parts are similar.
-
A structure that combines a motion-style encoder with a speaker classification loss to group the style embeddings of the same speaker (Mimic's auxiliary style classifier ↔ in this paper)
-
A method that aligns audio and motion representations with a CLIP-inspired bidirectional contrastive loss (Mimic's content contrastive loss ↔ in this paper)
-
A motivation experiment that trains a speaker classifier on motion sequences to show that motion contains speaker-specific style information (Mimic's Motivation section ↔ Sec. 3 and Fig. 2 in this paper)
The paper cites Mimic only as the source of the SCS metric (L406–407) and does not discuss the methodological differences from Mimic. The authors should clearly discuss how their method differs from Mimic. In addition, since Mimic extracts style from reference motion, it can also be applied to the setting of this paper by using source-language motion as the reference. In fact, Imitator, which is also a reference-based method, is included as a baseline. A performance comparison with Mimic should also be provided.
- Missing Citations and Related Work:
-
Missing citation for StreamingTalker (Yang et al., AAAI 2026), a recent work on speech-driven 3D facial animation.
-
SelfTalk is used as a baseline in Table 2 but is not included in the references.
-
MultiTalk, RAVDESS, and HDTF are compared with XLingo-12 in Table 1 but are not cited.
- Unclear Experimental Setting of the Baselines: The paper only describes the training setting of the proposed method (Sec. 5.2), and does not state whether the baselines were retrained on XLingo-12 or whether their publicly released checkpoints were used as is. If only the proposed method was trained on XLingo-12, its performance improvement could also be explained by training on more data from the same distribution as the evaluation data, rather than by the effect of the method itself. In addition, the proposed method uses 10 additional target-language utterances per speaker (L386) and uses HuBERT as an additional encoder besides Wav2Vec 2.0 (L389–391). To resolve this ambiguity, the authors should clearly state the following:
-
Whether the baselines were trained on the source- and target-language data of XLingo-12
-
Which pretrained audio encoders the baselines used
-
Whether target-language speech was also used to train the baselines
Questions for authors
- Which model was used for the visualization in Fig. 2, and how were the features extracted?
- All evaluations in this paper are performed only on the target-language test set, and the performance on the source language is not reported. However, the style-replacement consistency loss (Eq. 14) makes the outputs with the source-language style and the target-language style the same on the source-language speech content, so it also directly affects the motion generated with the source-language style. Compared with a model trained without them, does the final model trained with target-language speech and the related losses and $$\mathcal{L}_\text{swap}$) improve or degrade on a source-language test set?
- The paper evaluates only on XLingo-12, which the authors built. However, since the 3D motion in XLingo-12 is pseudo-GT estimated with EMOCA, it is necessary to check whether the performance of the proposed method holds on real 3D scan data. How does the method perform on VOCASET, which consists of high-quality 4D scans? I would also like to see the results on 3D-HDTF and TFHP, which are built by estimating FLAME parameters from videos, similar to XLingo-12. These datasets are monolingual, so the cross-lingual setting cannot be applied directly. However, they can show whether the personalization performance of the proposed method holds against existing methods on standard benchmarks.