acceptodds
Under review as a conference paper at ICLR 2027

Talk Like You: Imitating How You Speak in Real-Time Talking Head Generation

Abstract

In daily life, each person exhibits unique speaking habits, leading to subtle yet consistent lip-shape variations even when pronouncing the same word. Although recent talking head generation methods have achieved impressive visual fidelity and lip synchronization, they largely overlook user-specific customization, especially the motion patterns that characterize individual speaking habits. These habits are difficult to model and capture, as their motion patterns are highly fine-grained and often similar across individuals. As a result, many approaches produce overly uniform facial motions and fail to capture diverse, person-specific articulation patterns. To address this, we propose TalkLikeYou, an efficient framework that imitates how a target person speaks in talking head generation. Our method models habit in motion-space and achieves real-time performance through Flow Matching with only one sampling step during inference. We further adopt a two-stage imitation learning strategy to capture subtle distinctions between habits, allowing users to specify a target habit through either a preset style from the dataset or a reference video. In addition, we introduce a new metric PLAD that projects mouth motions onto representative articulation axes to evaluate imitation accuracy and generation diversity. Extensive experiments demonstrate that TalkLikeYou generates high-quality talking heads in real-time and significantly improves speaking habit imitation compared with prior methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Talking Head, Speaking Habit, Digital Human, Customized, Audio

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pHU9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TalkLikeYou, an audio-driven talking-head generation framework that aims to reproduce person-specific lip articulation patterns while maintaining lip synchronization. The framework combines a motion-space Flow Matching generator with a pretrained LivePortrait rendering pipeline. A two-stage training strategy aligns identity-specific one-hot embeddings with features extracted from reference motion sequences, supporting both preset and reference-based habit control. The paper also introduces Projected Lip-Articulation Distribution (PLAD), which compares motion histograms along three lip-articulation axes. The application is relevant, and the system demonstrates practical engineering value. However, the methodological contribution appears incremental, primarily integrating established components rather than introducing a substantially new modeling approach.

Critical strengths

Relevant personalization objective. Modeling person-specific articulation patterns addresses a meaningful limitation of talking-head systems that prioritize synchronization and visual quality. Supporting both preset habits and reference-video conditioning offers useful flexibility.

Practical generation efficiency. The reported 32.43 FPS at 512 × 512 resolution suggests potential for practical deployment. The combination of low-dimensional motion generation and pretrained rendering provides an efficient implementation.

Encouraging empirical results. The method reports competitive lip synchronization and visual quality, particularly on HDTF. The supplementary user study also favors the proposed method across the evaluated dimensions, providing some perceptual support beyond automatic metrics.

Critical weaknesses

Limited methodological novelty. The framework combines established techniques: motion-space generative modeling, direct clean-motion prediction, pretrained warping-based rendering, identity embeddings, and feature alignment. The two-stage training strategy is reasonable, but the paper does not sufficiently establish what new technical insight it contributes beyond this integration. In particular, motion-space Flow Matching has already been explored by FLOAT, while reference-based speaking-style control has been studied by StyleTalk and related methods. The current evidence supports an engineering contribution more strongly than a substantial methodological advance.

The distinction between speaking habits and speaking styles is insufficiently established. The paper emphasizes fine-grained, persistent articulation patterns rather than broader emotional or stylistic variation. However, assigning an identity-specific code does not by itself demonstrate that the learned representation isolates such habits. It may also encode identity-related geometry, expression tendencies, or dataset-specific characteristics. Stronger controlled experiments are needed to establish that the proposed representation captures a distinct behavioral factor.

PLAD has not been sufficiently validated as a habit-specific metric. Its marginal histograms can vary with phonetic content, speaking rate, emotion, and pause frequency. Longer sequences do not necessarily eliminate these confounds. Moreover, the descriptor discards temporal ordering: shuffling the frames leaves PLAD unchanged despite destroying motion dynamics. The metric may therefore capture coarse articulation statistics without adequately measuring person-specific speaking behavior. Its relationship to human judgments of habit similarity remains unclear.

The experimental protocol leaves important uncertainties. The paper uses HDTF and CelebV-HQ for both training and evaluation without clearly specifying identity-disjoint splits or the separation between reference and target clips. Consequently, generalization to unseen identities is difficult to assess. The habit-imitation comparisons and two-stage ablation also lack clearly reported aggregate statistics and uncertainty estimates.

Several evaluation and reproducibility details are missing. The construction of the PLAD projection axes, histogram settings, and scaling of the diversity scores are insufficiently specified. The user study does not clearly report the exact participant count, randomization and blinding procedures, or confidence intervals, and omits MimicTalk despite its relevance. Finally, FPS alone does not establish low-latency streaming performance without details on generation windows, audio lookahead, and end-to-end latency.

Questions for authors
  1. What is the principal methodological advance beyond combining existing motion-space generation, identity conditioning, feature alignment, and pretrained rendering? How does the approach differ technically from FLOAT combined with reference-based style conditioning?

  2. Can you provide controlled evidence that the learned representation captures persistent articulation habits independently of phonetic content, emotion, and facial geometry? Comparisons involving the same person across different utterances and different people speaking the same content would be particularly informative.

  3. Are the evaluation identities excluded from both training stages? Please specify the identity-level splits and how reference clips are separated from training and evaluation targets.

3 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer pHU9 - Part 1

We would like to thank the reviewer for their thorough assessment and valuable feedback on our submission. We are pleased that you recognized the core strengths of our work, particularly the importance of modeling person-specific articulation patterns, the practical benefits of efficient generation, and our competitive experimental results. We address your concerns and questions below.


1. Methodological Novelty (Q1 and W1)

We understand your concern that our approach builds on prior work. While we do use some existing techniques, we adapt them to better suit our task. For example, although methods such as FLOAT also use Flow Matching, we reparameterize the model to predict x₀ directly rather than velocity. This formulation enables one-step sampling while maintaining output quality, resulting in faster and more stable generation than FLOAT. Second, our two-stage training strategy follows the principle of first aligning features and then improving generalization, rather than being merely an engineering choice. We consider this design another contribution of our work. We would also like to emphasize that our primary focus is not on methodological novelty alone, but also on the task of speaking-habit imitation and its evaluation, which we consider more important.

2. Evidence for Speaking-Habit Control (Q2 and W2)

We understand your concern about the evidence for speaking-habit control. As shown in Fig. 10 (b) (Appendix D.4) and the table below, we compare recordings of the same speaker across different utterances with those of different speakers saying the same content generated by our method. The heatmap shows higher similarities within each speaker’s block along the diagonal than between speakers. This indicates that references from the same speaker yield more similar habitual motions despite differences in content, whereas those from different speakers produce distinct motion patterns even when the content is identical.

Comparison setting of audio content disentanglementHistogram IoUJS distance
Same speaker across different utterances0.600.06
Different speakers saying the same content0.270.24

Regarding other factors, we use relative expression deformations to capture individual motion patterns while reducing the influence of facial geometry. As shown in Fig. 11 (Appendix D.5), we observe more natural motions with relative expression generation than without it, suggesting that this formulation helps reduce the influence of geometric differences. Stable expressive tendencies are considered part of a person’s speaking habit, while training on extensive in-the-wild data in the second stage helps reduce dataset-specific biases. We further analyze the learned habit embeddings to assess how well they separate speaking habits from emotion, as shown in Fig. 10 (c) (Appendix D.5). The scatter plot shows that samples from the same speaker remain relatively close across emotions, while those from different speakers are more widely separated. Although surprised expressions are more challenging because sustained mouth opening can obscure habitual articulation, each speaker still forms a recognizable cluster, with less variation across emotions than between speakers. We plan to address this limitation through more effective emotion modeling in future work.

AuthorsOpenReview ↗

Rebuttal to Reviewer pHU9 - Part 2

3. Identity Exclusion during Evaluation (Q3 and W4)

We exclude all training identities from evaluation. Specifically, we use 380 identities from HDTF for training and reserve the remaining 20 for testing. CelebV-HQ is also split by identity, ensuring no overlap between the training and test sets. We further conduct experiments on unseen identities from another dataset, as shown in the table below (Tab. 4 in Appendix D.3), where our method outperforms the baselines in habit imitation, providing additional evidence of its ability to generalize to new speakers.

BaselineHistogram IoU ↑JS distance ↓User Study
StyleTalk0.280.271.1
PC-Talk0.360.142.7
MimicTalk0.500.113.0
TalkLikeYou (stage 1)0.480.122.7
TalkLikeYou (stage 2)0.610.064.5

4. Validation of PLAD (W3)

We understand your concern about the validity of PLAD. Our metric uses histogram features to characterize the distribution of lip articulations, which we believe reflects a person’s speaking habits. The results in Fig. 10 (a) (Appendix D.4) and table below show that our PLAD evaluates speaking-habit similarity regardless of the audio content. The Tab. 3 (Appendix D.6) and Tab. 4 (Appendix D.3) also support this view and show trends consistent with our user-study findings, providing further evidence of PLAD’s effectiveness.

Comparison setting of PLAD validationHistogram IoUJS distance
Same speaker across different utterances0.770.03
Different speakers saying the same content0.180.27

The PLAD focus on the frequency of particular articulation patterns rather than their temporal order, as these recurring tendencies help characterize how an individual speaks. We assume that, over sufficiently long sequences, speaking rate and pause frequency exhibit relatively stable statistical patterns, allowing the histograms to capture a broad range of articulatory behaviors. We therefore restrict our evaluation to long sequences, where this assumption is more likely to hold.

5. Additional Implementation Details (W5)

We understand your request for more implementation details. The three PLAD axes are based on blendshape parameter directions in FLAME [1] corresponding to vertical mouth opening, lateral lip widening, and forward protrusion during lip pursing. We manually vary these parameters and extract the resulting lip motions to obtain the projection axes. Our user study involved 23 participants. We omitted MimicTalk from this study because it requires separate training for each identity, making it difficult to generate a sufficient number of samples. For the speed evaluation, we use a generation window of 25, with 502 ms of audio lookahead and an end-to-end latency of 981 ms. We will release the code and the configurations used for our inference-speed evaluation upon publication.


Reference

[1] Li T, Bolkart T, Black M J, et al. Learning a model of facial shape and expression from 4D scans[J]. ACM Trans. Graph., 2017, 36(6): 194:1-194:17.


We are grateful for your thoughtful review and hope the clarifications above address your remaining concerns. Please let us know whether we have addressed your concerns and whether you have any other questions or requests that would help you raise your score in favor of acceptance.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer pHU9,

Please let us know whether our rebuttal has adequately addressed your remaining concerns or whether any further clarification would be helpful. We sincerely appreciate your time and consideration.

Thank you,

All Authors

Reviewer_Abk2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TalkLikeYou, a talking-head generation framework designed to imitate fine-grained, person-specific speaking habits. The method extracts motion-based speaking-habit features from reference videos or preset habit codes, maps them into a unified habit space through a two-stage imitation learning strategy, and then uses Flow Matching in motion space to generate facial motions that reflect the target speaking habits. The use of one-step sampling further enables real-time inference. In addition, the authors propose the PLAD metric to evaluate speaking-habit similarity and generation diversity.

Overall, the paper addresses a meaningful problem, and modeling personalized articulation patterns has potential practical value. However, the core framework mainly combines existing components, including motion extraction, attention-based conditional modeling, Flow Matching, and embedding alignment. Therefore, the main novelty lies in the task formulation, habit-conditioning strategy, and evaluation metric design, while the algorithmic innovation remains relatively limited. The authors should provide stronger evidence that the learned habit representation is genuinely disentangled from speaker identity, audio content, and facial appearance, and that it can generalize reliably across different identities and unseen speaking habits. I recommend adding further experiments to more convincingly support these claims.

Critical strengths
  1. The paper focuses on imitating fine-grained, person-specific speaking habits. This direction is beneficial for personalized digital humans and talking-head generation.

  2. The unified habit space allows users to specify target speaking habits through either preset codes or reference videos, providing a flexible control mechanism for personalized generation and offering practical value for real-world applications.

  3. The paper applies Flow Matching in motion space and achieves real-time generation through one-step sampling while maintaining relatively strong temporal coherence. The proposed PLAD metric evaluates speaking-habit similarity and generation diversity, providing a useful complement to traditional lip-synchronization metrics.

Critical weaknesses

The method mainly combines existing techniques, including motion extraction, attention-based conditional modeling, Flow Matching, and embedding alignment, with a focus on extracting habit-related information. Therefore, its contribution in terms of model innovation appears insufficient. At present, it is unclear whether the model has truly learned speaking habits or is instead capturing factors such as speaker identity, facial appearance, or audio content. I recommend adding cross-identity transfer experiments, evaluations on unseen speaking habits, and disentanglement studies to verify the effectiveness of the learned representation. The authors may also need to design more effective and targeted modules to better model and substantiate the proposed speaking-habit representation.

Questions for authors

As I mentioned in the weaknesses

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer Abk2

We would like to thank the reviewer for their thorough assessment and valuable feedback on our submission. We are glad that you found our work meaningful and recognized the benefits of speaking-habit imitation for talking-head generation, the flexibility offered by two alternative habit sources, our real-time generation capability, and the soundness of PLAD as a complement to lip-sync metrics. We address your concerns and questions below.


1. Extension of Existing Techniques

We understand your concern that our method builds on existing techniques. While we use designs such as Flow Matching and feature alignment, we adapt them to better suit our task. For example, we reparameterize the Flow Matching objective so that the model directly predicts the clean motion x₀. This allows our model to use one-step sampling while maintaining stable performance, as shown by the comparison with the original Flow Matching formulation in Fig.6 and Fig.12 (Appendix D.7) of the paper. We also adopt a two-stage strategy that combines feature alignment with subsequent training on an in-the-wild dataset to improve generalization. Our model is developed to serve the task of imitating how a person speaks, which we believe is both novel and meaningful.

2. Additional Experiments on Speaking-Habit Imitation

We understand your concern about whether the model truly captures speaking habits. Following your suggestion, we have conducted a disentanglement study of the relevant factors (Appendix D.5) and evaluated our method on unseen identities from another dataset (Appendix D.3). Our model operates in motion space and predicts relative motion, helping separate appearance from motion patterns. This formulation allows the model to focus on habitual articulation while reducing the influence of identity-specific visual features.

Regarding audio content, it primarily guides lip synchronization rather than defining speaking habits. As shown in Fig. 10 (b) (Appendix D.4) and the table below, we compare different speakers saying the same content and the same speaker across different utterances generated by our method. References from the same speaker yield more similar generated motions despite differences in content, whereas those from different speakers remain distinguishable even when the content is identical. These observations support the model’s ability to capture speaking habits beyond the content of the reference speech.

Comparison setting of audio content disentanglementHistogram IoU ↑JS distance ↓
Same speaker across different utterances0.600.06
Different speakers saying the same content0.270.24

We further examine factors such as facial geometry and emotion. We use relative expression generation to reduce the influence of geometric differences, such as variations in lip thickness. As shown in Fig. 11 (Appendix D.5), we observe more natural motions with relative expression generation than without it, suggesting that this formulation helps reduce the influence of facial geometry. We also visualize the learned habit embeddings across emotions in Fig. 10 (c) (Appendix D.5). Samples from the same speaker remain relatively close, while those from different speakers are more widely separated. Although surprised expressions remain challenging because sustained mouth opening can obscure habitual articulation, each speaker still forms a recognizable cluster. We plan to address this limitation in future work.

Finally, the experiments on unseen identities in the table below (Tab. 4 in Appendix D.3) show that our method outperforms the baselines in both PLAD metrics and user-study scores, providing further evidence of its ability to generalize to new speakers.

BaselineHistogram IoU ↑JS distance ↓User Study↑
StyleTalk0.280.271.1
PC-Talk0.360.142.7
MimicTalk0.500.113.0
TalkLikeYou (stage 1)0.480.122.7
TalkLikeYou (stage 2)0.610.064.5

3. More Effective Design

We designed a more targeted motion representation for articulation-habit generation by filtering out non-lip movements. As discussed above, we plan to introduce explicit emotion conditioning to make speaking-habit generation more responsive to emotional variations. Speaking motion can be understood as a combination of common motion patterns shared across speakers, person-specific habitual motion, and emotion-related motion. Our current model effectively learns person-specific habitual patterns through training on extensive real-world data. In future work, we will further disentangle emotion-related motion from habitual motion and model it explicitly.


 

We sincerely appreciate your thoughtful questions and the constructive nature of your feedback. Please let us know if there is anything else we can do to help you further improve your evaluation of our work.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer Abk2,

Please let us know if our rebuttal adequately addressed your concerns or if there is anything else we can clarify. We appreciate your time and consideration.

Thank you,

All Authors

Reviewer_G8sAOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TalkLikeYou, a two-stage talking-head framework that aims to imitate a target person's lip articulation ("speaking habits"). A DiT motion generator, trained with Flow Matching reparameterized to predict clean motion (x0x_0), produces 18-dimensional implicit-keypoint lip motion in one sampling step, which a frozen LivePortrait pipeline renders. The target habit is given either as a preset one-hot code or as a reference motion sequence. A two-stage schedule maps both into a shared habit space: one-hot training on HDTF with feature alignment, then reference-conditioned training on CelebV-HQ. The paper also introduces PLAD, which projects lip motion onto three articulation axes (opening, widening, pursing) and compares the resulting histograms with IoU and JS divergence.

Critical strengths
  1. Person-specific lip articulation is a relevant and under-explored aspect of talking-head personalization, and supporting both preset codes and reference videos is practical.
  2. Generating low-dimensional motion in one step and reusing a lightweight renderer gives real-time speed (32.43 FPS), a clear advantage over pixel-space diffusion models such as Hallo2 and Sonic.
  3. Standard lip-sync metrics do not capture personal articulation style, so an interpretable metric like PLAD is a worthwhile direction.
  4. Qualitative results cover stylized portraits, cross-habit comparisons, and a user study.
Critical weaknesses

I currently lean toward rejection, mainly because the habit-imitation claims rest on a metric whose validity is not shown, and because the main quantitative comparison may not be run under matched conditions.

  1. PLAD's validity is not established, yet the core habit claims depend on it. PLAD aggregates per-frame lip projections into histograms without controlling for phonetic content. The paper acknowledges that each frame is driven by phonetics but assumes, without evidence, that long sequences yield a stable, speaker-specific distribution. Two clips of the same speaker with different phoneme content could produce different histograms, while two speakers saying similar content could overlap. The three axes are described only through example words ("bus", "leave", "food"), with no formal construction, and PLAD is not checked against human judgments of habit similarity. Computing PLAD in the same LivePortrait motion space the generator is trained in may also favor the proposed method over pixel-space baselines. Since Fig. 4, Tab. 2, and Fig. 5 all rely on PLAD, this undermines the habit-imitation results.

  2. The protocol behind Table 1 is underspecified. On HDTF, TalkLikeYou reports FID 7.71 against 21.61–87.60 for the baselines, and an unusually high CSIM of 0.94. Sec. 3.1 and App. C.3 state that the pipeline also supports video dubbing, where only lip keypoints are edited and the face is stitched back into the original frame. If Table 1 used this setting while the baselines animate a single source image, the FID, FVD, and CSIM gains would be confounded by the input setting. Even in the single-image case, it is unclear whether cropping, resolution, and reference-frame selection are matched across methods. The gap also shrinks sharply on CelebV-HQ (FID 61.51 vs. 62.67 for Ditto), which is consistent with either a setting mismatch or identity overlap with the HDTF training data.

  3. Train/test identity overlap is unclear. Stage 1 assigns a one-hot code to each of the 380 HDTF identities and Stage 2 trains on CelebV-HQ, but the paper does not say whether the test identities in Tab. 1 and Tab. 2 are disjoint from either training set. If they overlap, the results do not show generalization to unseen habits, especially for one-hot conditioning.

  4. Habit-imitation evidence is thin. The comparison with StyleTalk, MimicTalk, and PC-Talk (Fig. 4) uses a single reference speaker, with no aggregate PLAD results across reference speakers. The diversity score of 71.14 in Tab. 2 cannot be a raw average JS divergence, which is bounded by ln⁡2\ln 2 (or 1), so some undisclosed scaling is applied. The baselines in Tab. 2 have no habit conditioning, so comparing them against randomly sampled habit features is questionable. Diversity reported without a realism check can also be inflated by noisy motion.

  5. Novelty is limited and the positioning is incomplete. Personalized speaking-style modeling has been studied in 2D (e.g., SAAS, AAAI 2024; StyleTalk++, TPAMI 2024) and in 3D (e.g., Mimic, AAAI 2024; MemoryTalker, ICCV 2025; Model See Model Do, SIGGRAPH 2025), and none of these are cited. More importantly, the paper does not clearly separate itself from PC-Talk (CVPR 2026), which it compares against. PC-Talk already uses implicit keypoints with a LivePortrait renderer, one-hot style conditioning, and lip deformation along opening, widening, and pursing axes. The style-vs-habit distinction in App. B.1 is stated conceptually but is not reflected in the architecture or the training objective. What remains new is one-step sampling, the two-stage schedule, and PLAD, none of which is cleanly isolated in experiments.

  6. The role of Flow Matching is unclear. With x0x_0 prediction and a single inference step, the generator is effectively a conditional regressor that takes noise as an extra input. Without a deterministic regression baseline using the same backbone, it is unclear what Flow Matching contributes. The ablation in Fig. 6 reports only lip-sync metrics, and its legend and axis labels are too small at normal reading size.

Questions for authors
  1. Can you show that PLAD separates speakers rather than phonetic content, e.g., that same-speaker/different-content distances are smaller than different-speaker/same-content distances, or that PLAD agrees with human judgments? How are the three axes constructed, what scaling is applied to the diversity score, and does the neutralization step in App. C.3 affect the histograms?
  2. Which input setting (single-image animation or video dubbing) was used for TalkLikeYou and for each baseline in Table 1? If they differ, can you report all methods under the same single-image protocol?
  3. Are the HDTF and CelebV-HQ test identities disjoint from training? Can you report reference-conditioned habit imitation on identities unseen in both stages, across multiple speakers?
  4. Can you (a) compare against the same backbone trained as a deterministic regressor, (b) report the velocity-prediction variants in Fig. 6 on quality and habit metrics rather than Sync-C alone, and (c) give a quantitative test-set ablation of the two-stage schedule?
  5. What are the concrete technical differences from PC-Talk in motion representation, rendering, habit conditioning, articulation control, and training?

Answers to Q1–Q3 are the most likely to change my score.

8 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer G8sA - Part 1

We would like to thank the reviewer for their thorough assessment and valuable feedback on our submission. We are pleased that you recognized the core strengths of our work, particularly the importance of person-specific lip articulation, our exploration of PLAD for habit evaluation, and the practical benefits of real-time inference and two alternative habit sources. We address your concerns and questions below.


1. Validation of PLAD (Q1 and W1)

We acknowledge that PLAD relies on the assumption that a speaker’s projected motion distributions remain stable over sufficiently long sequences. To examine this assumption, we have added experiments comparing the same speaker across different utterances and different speakers saying the same content with real video, as shown in table below and Fig. 10(a) (Appendix D.4). The heatmap shows higher histogram IoU for the same speaker despite changes in content, while similarities between different speakers remain lower even when the content is identical. These results suggest that histogram features from sufficiently long sequences can capture speaker-specific habitual patterns despite variations in speech content, supporting the validity of PLAD. Furthermore, the PLAD scores in Tab. 4 (Appendix D.3) show trends consistent with the user-study results, suggesting that PLAD reflects human judgments and provides an effective measure of habit-imitation quality.

Comparison setting of PLAD validationHistogram IoU ↑JS distance ↓
Same speaker across different utterances0.770.03
Different speakers saying the same content0.180.27

We define the three PLAD axes based on the blendshape parameter directions in FLAME [1] that correspond to vertical mouth opening, lateral lip widening, and forward protrusion during lip pursing. We manually vary these parameters and extract the resulting lip motions to obtain the projection axes. To ensure a fair comparison with image-space methods, we first generate talking head videos using each baseline and then evaluate the resulting videos with PLAD, rather than directly using their generated motion. We multiply the diversity score by 100 to make the differences easier to interpret and have corrected the corresponding equation in the paper. Regarding neutralization, we use it to place relative expression deformations in a common space across identities and provide a visible reference for the teeth, reducing the effects of imperfect keypoint disentanglement and improving temporal consistency. Although it does affect the histograms, its purpose is to improve lip synchronization and video quality rather than PLAD scores. Similar preprocessing is applied to all compared methods to ensure fairness.

2. Protocol for Quantitative Comparison (Q2 and W2)

All comparison results for TalkLikeYou are obtained using a single input image, consistent with the input settings of the other baselines. For evaluation, we download the checkpoints of the baseline methods and evaluate them on the target datasets, following the standard practice in talking-head generation. The smaller gains over the baselines on CelebV-HQ than on HDTF are mainly due to the challenging in-the-wild videos in that dataset, where occlusions and large head poses make generation more difficult.


Reference

[1] Li T, Bolkart T, Black M J, et al. Learning a model of facial shape and expression from 4D scans[J]. ACM Trans. Graph., 2017, 36(6): 194:1-194:17.

AuthorsOpenReview ↗

Rebuttal to Reviewer G8sA - Part 2

3. Identity Overlap between Training and Testing (Q3, W3, and W4)

We understand your concern about potential identity overlap and its impact on evaluation fairness. There is no identity overlap between training and testing in our evaluation. Specifically, we use 380 identities from HDTF for training, each represented by a one-hot code, and reserve the remaining 20 identities for evaluation. For CelebV-HQ, we split the dataset by identity to keep the training and test sets disjoint, ensuring a fair comparison. Our evaluation therefore already uses speaking habits from unseen identities as input. Additionally, we evaluate habit imitation on speakers from another dataset who were not included in either training stage, as shown in table below (Tab. 4 in Appendix D.3). Our method shows a clear advantage over the relevant baselines. In particular, the second-stage model achieves substantial improvements over the first-stage model, further supporting the effectiveness of our two-stage habit-imitation learning strategy.

BaselineHistogram IoU ↑JS distance ↓User Study
StyleTalk0.280.271.1
PC-Talk0.360.142.7
MimicTalk0.500.113.0
TalkLikeYou (stage 1)0.480.122.7
TalkLikeYou (stage 2)0.610.064.5

4. Clarification of Technical Novelty and Flow Matching (Q5, W5, and W6)

We understand your concern about the novelty of our work and will revise the relevant citations. Compared with PC-Talk, our method adopts Flow Matching and a two-stage training schedule, yielding substantial performance gains. During training, we reparameterize the model to predict the clean motion x₀ directly while retaining the noise-based formulation and interpolation between noise and clean motion at timestep t. This formulation also supports multi-step sampling, though we use a single step to reduce inference time while maintaining high generation quality. Note that our model retains randomness through the sampled noise even with one-step sampling, distinguishing it from a deterministic regressor. We have conducted several experiments, as detailed in our response to the next point. Furthermore, while PC-Talk relies on manual editing along several axes to improve its initial imitation results, our method directly generates motions that reflect the target articulation patterns. We use these axes only for evaluation, rather than as controls for adjusting the generated videos. The experiments in Fig. 5 and 6 demonstrate the effectiveness of our reparameterization formulation and two-stage training strategy. Meanwhile, the video results further show that our method reproduces mouth movements more faithfully while maintaining natural and coherent motion.

AuthorsOpenReview ↗

Rebuttal to Reviewer G8sA - Part 3

5. Additional Experiments (Q4,W4 and W6)

Following your suggestion, we have conducted several experiments to provide further support for our claims.

(a) We compare our method with the same backbone trained as a deterministic regressor, as shown in the table below. We evaluate lip synchronization, visual quality, and PLAD similarity across multiple speakers. Our method achieves higher Sync-C and PLAD similarity, supporting the benefits of our one-step Flow Matching formulation. These findings suggest that our method benefits from generative modeling while retaining inference efficiency comparable to a deterministic regressor.

MethodSync-C ↑NIQE ↓PLAD similarity ↑
Deterministic regression (same backbone)8.0212.290.47
TalkLikeYou8.6612.270.60

(b) We compare our formulation with velocity-prediction variants and examine the effect of reparameterization. Image-quality metrics remain largely consistent across variants, as they share the same rendering pipeline and differ only in motion generation. We use JS divergence to assess habit-imitation quality, as shown in Fig. 12 (Appendix D.7). Flow Matching consistently achieves lower divergence than DDIM across sampling steps, and reparameterization substantially improves both formulations. These results support the effectiveness of combining Flow Matching and reparameterization for efficient and faithful speaking-habit imitation.

(c) We compare generation diversity across habit-imitation methods using a fixed random seed. We omitted this comparison from the original submission because not all methods provide an editable habit space for random sampling; some support only reference-based imitation. To accommodate these methods, we randomly sample unseen identities and use their videos as reference inputs for diversity evaluation. As shown in the table below, our method achieves higher diversity, indicating that it can generate a broader range of distinct speaking habits.

MetricStyleTalkMimicTalkPC-TalkTalkLikeYou
Diversity (×100) ↑31.0959.1850.1368.37

The test-set ablation of the two-stage training schedule is presented in our response to Point 3, with results in Tab. 4 (Appendix D.3). Together with the PLAD validation discussed above, these experiments separately assess our one-step sampling formulation, two-stage training strategy, and evaluation metric, providing further evidence of their effectiveness.


We are grateful for your thoughtful review and hope the clarifications above address your remaining concerns. Please let us know whether we have addressed your concerns and whether you have any other questions or requests that would help you raise your score in favor of acceptance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers (G8sA, Abk2, pHU9)

We sincerely thank reviewers G8sA, Abk2, and pHU9 for their thoughtful assessment and constructive feedback. We appreciate the reviewers’ recognition of TalkLikeYou’s strengths, particularly its focus on imitating person-specific articulation behaviors, a meaningful yet underexplored direction in talking-head generation. The reviewers also highlighted the framework’s practical value, noting its real-time inference speed and support for two alternative habit sources. In particular, reviewers G8sA and Abk2 noted our effort to evaluate speaking habits using PLAD, while reviewer pHU9 found the experimental results encouraging.

In response to concerns such as PLAD validation, experimental settings, and the use of existing techniques, we have added further experiments and expanded the corresponding explanations in the paper, with the additions highlighted in blue, as well as in the responses below. The additional experiments include:

  • Following the suggestions of reviewers G8sA and pHU9, we assess the validity of PLAD by comparing different speakers saying the same content and the same speaker across different utterances (Appendix D.4, Fig. 10(a)).
  • Following the suggestions of reviewers G8sA, Abk2, and pHU9, we evaluate generalization to unseen speaking habits using data from another dataset (Appendix D.3, Tab. 4).
  • Following the suggestions of reviewer G8sA, we add diversity comparisons with related methods, a deterministic regression baseline using the same backbone, PLAD evaluation of velocity-prediction variants (Appendix D.7, Fig. 12), and a quantitative ablation of the two-stage training strategy (Appendix D.3, Tab. 4).
  • Following the suggestions of reviewers Abk2 and pHU9, we conduct a disentanglement study examining the effects of identity, audio content, emotion, and facial geometry (see Appendix D.5, Fig. 10(b) (c) and 11).

We will address each reviewer’s questions and concerns through detailed, point-by-point responses below. We also correct several typos and revise the citations in the paper. To complement these explanations, we also encourage reviewers to watch the demo and comparison videos in our supplementary ZIP file, which offer a more direct view of the improvements in overall quality and speaking-habit imitation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.