acceptodds
Withdrawn from ICLR 2027

XLingoTalker: Cross-Lingual Personalized Speech-Driven 3D Facial Animation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Cross-lingual personalized 3D facial animation aims to generate speech-synchronized facial motion in a new language while preserving a speaker's characteristic articulation patterns. Existing methods typically rely on language-matched speech-motion pairs and therefore cannot address cases where target-language motion is unavailable. We formulate this problem as weakly supervised cross-lingual personalized speech-driven 3D facial animation, where paired speech-motion data are available only in a source language, together with unpaired target-language speech from the same speaker. The key challenge is to separate speaker-specific motion style from language-dependent acoustic variation. We observe that audio-derived style representations vary substantially across languages, whereas motion-derived speaker representations remain more consistent. Motivated by this finding, we propose XLingoTalker, which grounds audio-derived style in a shared audio-motion space. A style-guided multi-head mixture-of-experts decoder maps the grounded representation to personalized facial dynamics, while a style-replacement consistency objective leverages unpaired target-language speech to preserve speaker-specific motion across languages. We further introduce XLingo-12, the first multilingual 3D talking-face dataset featuring controlled bilingual recordings from 12 speakers for fixed-identity cross-lingual evaluation. Experiments show that XLingoTalker better preserves personalized facial dynamics while maintaining speech-motion synchronization.

Primary area:
datasets and benchmarks
Keywords:
Speech-Driven 3D Facial Animation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nWG5OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a cross-lingual personalized speech-driven 3D facial animation model that preserves the speaker-specific motion style when driven by target-language speech, even without target-language motion. Previous studies have primarily used audio-derived style representations or reference motion to reflect the speaker's style. However, audio-derived representations have limitations in that the inferred style changes even for the same speaker when the language changes, as the speaker's style is entangled with language-dependent acoustic variation. Reference-based representations have limitations in that they are influenced by sample-specific factors such as emotion, speaking rate, and local expression. To overcome these limitations, the authors propose a method to learn a shared audio-motion style space and to regularize style representations extracted from source- and target-language speech based on source-language motion. Through various experiments, the authors demonstrate that the proposed method is effective in terms of motion accuracy, temporal consistency, and speaker-specific style preservation.

Critical strengths
  • Validity of the problem setting: The motivation that the speaker-specific motion style should be preserved even when the language changes is valid. Target-language speech can be collected relatively easily, whereas 3D facial motion requires specialized capture equipment and substantial post-processing (L041–043). Considering this, it is good that the authors propose a weakly supervised learning that uses only source-language speech-motion pairs and target-language speech.

  • Performance in bidirectional evaluation: In both English-to-non-English and non-English-to-English transfer directions, the proposed method shows the best performance across all seven metrics (LVE, FVE, Lip-max, LDTW, FDTW, FID, SCS) compared to five baselines (Table 2). The bidirectional evaluation shows that the performance of the proposed method does not depend on a specific source language.

  • Intuitive visualization through demo videos: In the demo videos in the supplementary materials, the results of GT, the proposed method, and five baselines can be compared side by side with error heatmaps, allowing for an intuitive view of the differences in results in both transfer directions.

Critical weaknesses
  1. Overlap with Mimic and Missing Comparison: The proposed method has many methodological overlaps with Mimic (Fu et al., AAAI 2024). Specifically, the following three parts are similar.
  • A structure that combines a motion-style encoder with a speaker classification loss to group the style embeddings of the same speaker (Mimic's auxiliary style classifier ↔ Lm-cls\mathcal{L}_{\text{m-cls}} in this paper)

  • A method that aligns audio and motion representations with a CLIP-inspired bidirectional contrastive loss (Mimic's content contrastive loss ↔ Lalign\mathcal{L}_{\text{align}} in this paper)

  • A motivation experiment that trains a speaker classifier on motion sequences to show that motion contains speaker-specific style information (Mimic's Motivation section ↔ Sec. 3 and Fig. 2 in this paper)

The paper cites Mimic only as the source of the SCS metric (L406–407) and does not discuss the methodological differences from Mimic. The authors should clearly discuss how their method differs from Mimic. In addition, since Mimic extracts style from reference motion, it can also be applied to the setting of this paper by using source-language motion as the reference. In fact, Imitator, which is also a reference-based method, is included as a baseline. A performance comparison with Mimic should also be provided.

  1. Missing Citations and Related Work:
  • Missing citation for StreamingTalker (Yang et al., AAAI 2026), a recent work on speech-driven 3D facial animation.

  • SelfTalk is used as a baseline in Table 2 but is not included in the references.

  • MultiTalk, RAVDESS, and HDTF are compared with XLingo-12 in Table 1 but are not cited.

  1. Unclear Experimental Setting of the Baselines: The paper only describes the training setting of the proposed method (Sec. 5.2), and does not state whether the baselines were retrained on XLingo-12 or whether their publicly released checkpoints were used as is. If only the proposed method was trained on XLingo-12, its performance improvement could also be explained by training on more data from the same distribution as the evaluation data, rather than by the effect of the method itself. In addition, the proposed method uses 10 additional target-language utterances per speaker (L386) and uses HuBERT as an additional encoder besides Wav2Vec 2.0 (L389–391). To resolve this ambiguity, the authors should clearly state the following:
  • Whether the baselines were trained on the source- and target-language data of XLingo-12

  • Which pretrained audio encoders the baselines used

  • Whether target-language speech was also used to train the baselines

Questions for authors
  1. Which model was used for the visualization in Fig. 2, and how were the features extracted?
  2. All evaluations in this paper are performed only on the target-language test set, and the performance on the source language is not reported. However, the style-replacement consistency loss (Eq. 14) makes the outputs with the source-language style and the target-language style the same on the source-language speech content, so it also directly affects the motion generated with the source-language style. Compared with a model trained without them, does the final model trained with target-language speech and the related losses Lalign,t\mathcal{L}_{\text{align,t}} and $$\mathcal{L}_\text{swap}$) improve or degrade on a source-language test set?
  3. The paper evaluates only on XLingo-12, which the authors built. However, since the 3D motion in XLingo-12 is pseudo-GT estimated with EMOCA, it is necessary to check whether the performance of the proposed method holds on real 3D scan data. How does the method perform on VOCASET, which consists of high-quality 4D scans? I would also like to see the results on 3D-HDTF and TFHP, which are built by estimating FLAME parameters from videos, similar to XLingo-12. These datasets are monolingual, so the cross-lingual setting cannot be applied directly. However, they can show whether the personalization performance of the proposed method holds against existing methods on standard benchmarks.
Reviewer_8CQwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies cross-lingual personalized 3D facial animation using paired source-language speech and motion, together with a small amount of unpaired target-language speech from the same speakers. The authors propose XLingoTalker, a motion-grounded framework that integrates cross-lingual audio-motion style alignment, style-guided multi-head mixture-of-experts decoding, and style-replacement consistency for robust personalization across languages. The authors also introduce XLingo-12, containing 4,800 bilingual clips from 12 speakers, with facial motion estimated using EMOCA. They conduct experiments to validate the effectiveness of the proposed method in motion accuracy, temporal consistency, and speaker-specific style preservation.

Critical strengths
  1. The paper considers a useful setting: learning personalized facial animation across languages when target-language speech is available but paired facial motion is not. This makes the task relevant to practical applications, since collecting facial motion for every target language is costly.
  2. The paper is well-written and easy to follow.
Critical weaknesses
  1. It is unclear how the baselines use the available target-language speech. Section 5.2 specifies ten target-language utterances per speaker for motion-free adaptation, but does not describe how this adaptation is achieved for each baseline. Since removing target-language audio worsens performance in Table 3, the authors should clarify whether all baseline methods in Table 2 have access to the same training data and how the baselines are adapted.
  2. The facial motion in XLingo-12 is estimated from videos using EMOCA, but the paper does not report an assessment of the reconstruction quality. This makes the reliability of the dataset less convincing.
  3. XLingo-12 contains only 12 speakers, all seen during training, which limits the generalization ability of the proposed method. Each non-English language is also represented by a single speaker, confounding language and speaker identity. More speakers per language and evaluation on unseen speakers would strengthen the conclusions.
Questions for authors

See weakness.

Reviewer_3XkSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces weakly supervised cross-lingual personalized speech-driven 3D facial animation. For each speaker, paired speech–motion data exist only in a source language, plus a small amount of unpaired target-language speech. The goal is to animate target-language speech while keeping the speaker's articulation style. XLingoTalker has three components. (1) It pulls audio style embeddings from both languages toward speaker anchors derived from source-language motion, using a CLIP-style contrastive loss plus motion speaker classification. (2) It decodes with a multi-head MoE whose routing mixes a content term and a style term. (3) It adds a style-replacement consistency loss: source content decoded with source- or target-language style from the same speaker should give the same motion. The paper also introduces XLingo-12 (12 speakers, each with 200 English and 200 non-English utterances, motion estimated by EMOCA) and reports gains over five baselines on all seven metrics in both transfer directions.

Critical strengths
  1. The setting is practical. Target-language speech is cheap and target-language 3D motion is not, and the formulation is clearly distinct from both fully supervised multilingual animation and naive source-to-target transfer.
  2. Each component has a clear role, and the ablations (Table 3) and routing visualizations (App. A) show that each one changes the output.
  3. The experiments cover both transfer directions, a target-data sweep (0–20 utterances), and a user study, with consistent results.
Critical weaknesses

I lean toward rejection. The components may well help, but the experiments do not separate the claimed benefit, language-invariant motion style, from two simpler explanations: knowing who is speaking, and a stronger backbone.

W1. Simpler known-speaker conditioning is not compared, and identity shortcuts are uncontrolled. In XLingo-12 each non-English language is spoken by exactly one speaker, so in the English → non-English direction the target language identifies the speaker. The reverse direction removes this shortcut on the target side, but source motion still maps one-to-one to speakers, and the alignment loss defines positives purely by speaker identity. The experiments therefore cannot rule out that the model mainly learns speaker-discriminative codes. More importantly, every test speaker has source-language motion in training, so two simpler controls are fully available: (a) the same encoder and decoder with a learned speaker-ID embedding in place of zaz^a, and (b) a style code extracted from the speaker's existing source-language motion and cached for inference, as in reference-motion methods such as DiffPoseTalk. Neither needs target-language motion, and the paper should show an advantage over both. Including several speakers who share one non-English language would also separate language from identity.

W2. Most of the gain appears before any target-language adaptation. In English → non-English, LVE is 0.2833 for MemoryTalker, 0.2504 for XLingoTalker without target-language audio, and 0.2270 for the full model. About 58% of the improvement over the strongest baseline therefore exists without adaptation, and every ablated variant in Table 3 beats every baseline. The zero-target variant still has source-side alignment and style routing, but the ablations do not separate the contributions of the audio encoders (wav2vec 2.0 + HuBERT, checkpoints unspecified), the MH-MoE decoder, and the style supervision. The baselines' encoders, splits, and target-data budgets are not described either. Encoder-matched baselines, or the proposed components added to a baseline backbone, would make Table 2 interpretable. Separately, removing L_align worsens LVE by 7.8% even though λ_align = 5×10⁻⁴. Since this result is a key argument for the component, please explain it (e.g., the gradient magnitude relative to the reconstruction losses) or show over a few seeds that the gap exceeds run-to-run variation.

W3. The motivating comparison in Figure 2 uses mismatched supervision. The motion features come from a classifier trained with identity labels on these 12 speakers' source-language motion, while the paper does not say whether the PESTalk audio embeddings were adapted to XLingo-12 at all. Speaker clustering in the motion features largely follows from that training objective. A matched control would train the same kind of speaker classifier on source-language audio and apply it to target-language audio. Even if it transfers well, that would not show that audio captures motion style, but it would show whether the gap in Fig. 2 is an artifact of the different supervision.

W4. Style preservation is not directly measured. Ground truth is estimated by EMOCA, so per-speaker estimation errors (from appearance, viewpoint, or recording conditions) can look like style. SCS is computed from a speaker-classifier embedding, so it is likely most sensitive to speaker-discriminative cues, which is close to what L_m-cls and L_align optimize, even with separate parameters. SCS is reported only for the zero-target variant (Table 4), not for the other ablations in Table 3, so the effect of each component on style preservation is not shown. In the user study, raters saw the GT motion and error heatmaps beside each result, which makes the task closer to reference matching than to an independent style judgment. Two baselines were dropped as "visibly weaker", yet CodeTalker's English → non-English SCS (0.8655) is higher than that of the included Imitator (0.8361) and SelfTalk (0.7820).

W5. The dataset description overstates control. The abstract describes "controlled bilingual recordings" and the introduction "synchronized speech-motion recordings", but Sec. 5.1 states that motion is estimated by EMOCA from publicly accessible videos. The paper does not report the video sources, whether each speaker's English and non-English clips come from comparable sessions, reconstruction quality, or consent and licensing for releasing data of 12 identifiable people. Uncontrolled sources could confound both Fig. 2(a) and the style evaluation. Table 1 also marks VOCASET, BIWI, and MultiTalk ✗ for "paired data", although all three pair speech with 3D motion (pseudo-3D for MultiTalk). The column presumably means same-speaker bilingual pairs, and the "first multilingual 3D talking-face dataset" claim should be narrowed accordingly.

Positioning. MultiTalk already models multilingual speech-driven 3D animation, and Polyglot (Nocentini et al., 2026) combines language embeddings with reference-motion speaker style and a style-preservation loss. Both assume paired multilingual supervision, so they do not undercut the weakly supervised setting, but the paper should discuss them and make the distinction explicit.

Questions for authors

Answers to Q1–Q3 are the most likely to change my score.

  1. With the same encoder, decoder, and data, how does XLingoTalker compare with (a) a learned speaker-ID embedding and (b) a cached style code from each speaker's source-language motion?
  2. Can you add SCS, and ideally a style measure not based on speaker classification, to all ablations in Table 3, and report encoder-matched baselines?
  3. For Fig. 2, how does a speaker classifier trained on source-language audio behave on target-language audio? Can you report within-speaker cross-language versus between-speaker distances for both modalities?
  4. Can you evaluate several speakers who share the same non-English language?
  5. Where do the XLingo-12 videos come from? Are each speaker's two languages recorded under comparable conditions, and what consent and licensing apply to the release?
  6. What does the user study show without GT and error heatmaps, and how were the included baselines chosen?

If the rebuttal shows that the motion-grounded audio style clearly outperforms both alternatives in Q1 under the same backbone, and the gains hold against encoder-matched baselines, I would consider raising my score. Conversely, if these simpler alternatives perform on par with the full model, the paper's central mechanism would lose most of its support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.