- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents TalkLikeYou, an audio-driven talking-head generation framework that aims to reproduce person-specific lip articulation patterns while maintaining lip synchronization. The framework combines a motion-space Flow Matching generator with a pretrained LivePortrait rendering pipeline. A two-stage training strategy aligns identity-specific one-hot embeddings with features extracted from reference motion sequences, supporting both preset and reference-based habit control. The paper also introduces Projected Lip-Articulation Distribution (PLAD), which compares motion histograms along three lip-articulation axes. The application is relevant, and the system demonstrates practical engineering value. However, the methodological contribution appears incremental, primarily integrating established components rather than introducing a substantially new modeling approach.
Critical strengths
Relevant personalization objective. Modeling person-specific articulation patterns addresses a meaningful limitation of talking-head systems that prioritize synchronization and visual quality. Supporting both preset habits and reference-video conditioning offers useful flexibility.
Practical generation efficiency. The reported 32.43 FPS at 512 × 512 resolution suggests potential for practical deployment. The combination of low-dimensional motion generation and pretrained rendering provides an efficient implementation.
Encouraging empirical results. The method reports competitive lip synchronization and visual quality, particularly on HDTF. The supplementary user study also favors the proposed method across the evaluated dimensions, providing some perceptual support beyond automatic metrics.
Critical weaknesses
Limited methodological novelty. The framework combines established techniques: motion-space generative modeling, direct clean-motion prediction, pretrained warping-based rendering, identity embeddings, and feature alignment. The two-stage training strategy is reasonable, but the paper does not sufficiently establish what new technical insight it contributes beyond this integration. In particular, motion-space Flow Matching has already been explored by FLOAT, while reference-based speaking-style control has been studied by StyleTalk and related methods. The current evidence supports an engineering contribution more strongly than a substantial methodological advance.
The distinction between speaking habits and speaking styles is insufficiently established. The paper emphasizes fine-grained, persistent articulation patterns rather than broader emotional or stylistic variation. However, assigning an identity-specific code does not by itself demonstrate that the learned representation isolates such habits. It may also encode identity-related geometry, expression tendencies, or dataset-specific characteristics. Stronger controlled experiments are needed to establish that the proposed representation captures a distinct behavioral factor.
PLAD has not been sufficiently validated as a habit-specific metric. Its marginal histograms can vary with phonetic content, speaking rate, emotion, and pause frequency. Longer sequences do not necessarily eliminate these confounds. Moreover, the descriptor discards temporal ordering: shuffling the frames leaves PLAD unchanged despite destroying motion dynamics. The metric may therefore capture coarse articulation statistics without adequately measuring person-specific speaking behavior. Its relationship to human judgments of habit similarity remains unclear.
The experimental protocol leaves important uncertainties. The paper uses HDTF and CelebV-HQ for both training and evaluation without clearly specifying identity-disjoint splits or the separation between reference and target clips. Consequently, generalization to unseen identities is difficult to assess. The habit-imitation comparisons and two-stage ablation also lack clearly reported aggregate statistics and uncertainty estimates.
Several evaluation and reproducibility details are missing. The construction of the PLAD projection axes, histogram settings, and scaling of the diversity scores are insufficiently specified. The user study does not clearly report the exact participant count, randomization and blinding procedures, or confidence intervals, and omits MimicTalk despite its relevance. Finally, FPS alone does not establish low-latency streaming performance without details on generation windows, audio lookahead, and end-to-end latency.
Questions for authors
-
What is the principal methodological advance beyond combining existing motion-space generation, identity conditioning, feature alignment, and pretrained rendering? How does the approach differ technically from FLOAT combined with reference-based style conditioning?
-
Can you provide controlled evidence that the learned representation captures persistent articulation habits independently of phonetic content, emotion, and facial geometry? Comparisons involving the same person across different utterances and different people speaking the same content would be particularly informative.
-
Are the evaluation identities excluded from both training stages? Please specify the identity-level splits and how reference clips are separated from training and evaluation targets.
3 comments
Rebuttal to Reviewer pHU9 - Part 2
3. Identity Exclusion during Evaluation (Q3 and W4)
We exclude all training identities from evaluation. Specifically, we use 380 identities from HDTF for training and reserve the remaining 20 for testing. CelebV-HQ is also split by identity, ensuring no overlap between the training and test sets. We further conduct experiments on unseen identities from another dataset, as shown in the table below (Tab. 4 in Appendix D.3), where our method outperforms the baselines in habit imitation, providing additional evidence of its ability to generalize to new speakers.
| Baseline | Histogram IoU ↑ | JS distance ↓ | User Study |
|---|---|---|---|
| StyleTalk | 0.28 | 0.27 | 1.1 |
| PC-Talk | 0.36 | 0.14 | 2.7 |
| MimicTalk | 0.50 | 0.11 | 3.0 |
| TalkLikeYou (stage 1) | 0.48 | 0.12 | 2.7 |
| TalkLikeYou (stage 2) | 0.61 | 0.06 | 4.5 |
4. Validation of PLAD (W3)
We understand your concern about the validity of PLAD. Our metric uses histogram features to characterize the distribution of lip articulations, which we believe reflects a person’s speaking habits. The results in Fig. 10 (a) (Appendix D.4) and table below show that our PLAD evaluates speaking-habit similarity regardless of the audio content. The Tab. 3 (Appendix D.6) and Tab. 4 (Appendix D.3) also support this view and show trends consistent with our user-study findings, providing further evidence of PLAD’s effectiveness.
| Comparison setting of PLAD validation | Histogram IoU | JS distance |
|---|---|---|
| Same speaker across different utterances | 0.77 | 0.03 |
| Different speakers saying the same content | 0.18 | 0.27 |
The PLAD focus on the frequency of particular articulation patterns rather than their temporal order, as these recurring tendencies help characterize how an individual speaks. We assume that, over sufficiently long sequences, speaking rate and pause frequency exhibit relatively stable statistical patterns, allowing the histograms to capture a broad range of articulatory behaviors. We therefore restrict our evaluation to long sequences, where this assumption is more likely to hold.
5. Additional Implementation Details (W5)
We understand your request for more implementation details. The three PLAD axes are based on blendshape parameter directions in FLAME [1] corresponding to vertical mouth opening, lateral lip widening, and forward protrusion during lip pursing. We manually vary these parameters and extract the resulting lip motions to obtain the projection axes. Our user study involved 23 participants. We omitted MimicTalk from this study because it requires separate training for each identity, making it difficult to generate a sufficient number of samples. For the speed evaluation, we use a generation window of 25, with 502 ms of audio lookahead and an end-to-end latency of 981 ms. We will release the code and the configurations used for our inference-speed evaluation upon publication.
Reference
[1] Li T, Bolkart T, Black M J, et al. Learning a model of facial shape and expression from 4D scans[J]. ACM Trans. Graph., 2017, 36(6): 194:1-194:17.
We are grateful for your thoughtful review and hope the clarifications above address your remaining concerns. Please let us know whether we have addressed your concerns and whether you have any other questions or requests that would help you raise your score in favor of acceptance.
Official Comment by Authors
Dear Reviewer pHU9,
Please let us know whether our rebuttal has adequately addressed your remaining concerns or whether any further clarification would be helpful. We sincerely appreciate your time and consideration.
Thank you,
All Authors
Rebuttal to Reviewer pHU9 - Part 1
We would like to thank the reviewer for their thorough assessment and valuable feedback on our submission. We are pleased that you recognized the core strengths of our work, particularly the importance of modeling person-specific articulation patterns, the practical benefits of efficient generation, and our competitive experimental results. We address your concerns and questions below.
1. Methodological Novelty (Q1 and W1)
We understand your concern that our approach builds on prior work. While we do use some existing techniques, we adapt them to better suit our task. For example, although methods such as FLOAT also use Flow Matching, we reparameterize the model to predict x₀ directly rather than velocity. This formulation enables one-step sampling while maintaining output quality, resulting in faster and more stable generation than FLOAT. Second, our two-stage training strategy follows the principle of first aligning features and then improving generalization, rather than being merely an engineering choice. We consider this design another contribution of our work. We would also like to emphasize that our primary focus is not on methodological novelty alone, but also on the task of speaking-habit imitation and its evaluation, which we consider more important.
2. Evidence for Speaking-Habit Control (Q2 and W2)
We understand your concern about the evidence for speaking-habit control. As shown in Fig. 10 (b) (Appendix D.4) and the table below, we compare recordings of the same speaker across different utterances with those of different speakers saying the same content generated by our method. The heatmap shows higher similarities within each speaker’s block along the diagonal than between speakers. This indicates that references from the same speaker yield more similar habitual motions despite differences in content, whereas those from different speakers produce distinct motion patterns even when the content is identical.
Regarding other factors, we use relative expression deformations to capture individual motion patterns while reducing the influence of facial geometry. As shown in Fig. 11 (Appendix D.5), we observe more natural motions with relative expression generation than without it, suggesting that this formulation helps reduce the influence of geometric differences. Stable expressive tendencies are considered part of a person’s speaking habit, while training on extensive in-the-wild data in the second stage helps reduce dataset-specific biases. We further analyze the learned habit embeddings to assess how well they separate speaking habits from emotion, as shown in Fig. 10 (c) (Appendix D.5). The scatter plot shows that samples from the same speaker remain relatively close across emotions, while those from different speakers are more widely separated. Although surprised expressions are more challenging because sustained mouth opening can obscure habitual articulation, each speaker still forms a recognizable cluster, with less variation across emotions than between speakers. We plan to address this limitation through more effective emotion modeling in future work.