acceptodds
Under review as a conference paper at ICLR 2027

Behave Yourself! Personalizing Talking Heads Beyond Appearance

Abstract

Recent video generation models enable appearance-based personalization by conditioning generation on reference images of a specific person. Yet a person's identity is not fully captured by appearance alone. It also includes characteristic behavior, such as distinctive head movements one makes while speaking or listening, which is not directly exposed through static reference images. In this work, we introduce a spatio-temporal identity encoder that aggregates a variable number of reference videos into a fixed number of identity tokens, capturing both visual details revealed across frames as well as the subject's characteristic behavior. These identity tokens condition a frozen audio-driven video diffusion transformer through dedicated cross-attention layers, enabling generation with new speech while preserving the subject's appearance and characteristic behavior. A three-stage training procedure first adapts the video encoder to talking-head videos through self-supervised training. It then establishes identity conditioning using the target clip as reference, before switching to different reference clips of the same subject. Through qualitative and quantitative experiments, we demonstrate that our method yields more authentic, person-specific videos compared to an appearance-only baseline.

Primary area:
generative models
Keywords:
Diffusion models, Video models

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_kESHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper conditions a frozen talking-video diffusion model on multiple reference clips. A domain-adapted video encoder and a query-based aggregator compress these clips into a fixed set of identity tokens, which enter the generator through additional cross-attention. Training first uses the target clip as its own reference and then switches to non-overlapping reference clips of the same person. The intended benefit is to capture characteristic expressions and movements as well as appearance.

Critical strengths

Personalizing behavior is a useful goal, and the method improves expression and pose distribution distances over the image-reference baseline. The fixed number of conditioning tokens is a sensible design for multiple video references. The paper documents its architecture, preprocessing and evaluation in considerable detail, attributes its use of Q-Former and IP-Adapter, and includes relevant talking-avatar comparisons and a human study. It also clearly labels the preceding-ground-truth-clip setting as an oracle.

Critical weaknesses

1. The evaluation does not separate a person's characteristic behavior from the circumstances of a particular recording. Appendix B states that cross-clip references are other segments of the same longer recording. Removing temporal overlap does not remove shared mood, conversation, posture, clothing or camera setup. Better matching the target under this protocol is useful, but it does not establish that the representation captures behavioral identity that persists across sessions.

The metrics do not resolve this distinction. E-FID and P-FID compare distributions of individual frames, without measuring their temporal order. P-FID includes translation, and Mesh-FID includes facial geometry, so these scores can also respond to framing and appearance. The DINOv3 comparison changes the encoder and its pretraining, rather than isolating temporal information in the same encoder. The human study supports the quality of Non-Direct + Ours, but it does not test whether the behavior transfers across sessions or remains identifiable after appearance is controlled. I would like to see cross-session references, temporally shuffled or repeated-frame references with the same encoder, and a test of whether observers can identify a person's behavior across contexts. This matters because behavioral identity is the main claim distinguishing the paper from richer visual conditioning, including the already-cited TAVR [1].

2. Important ablations and metrics are not controlled well enough to support the attribution. Appendix C.1 trains the same-clip-only and cross-clip-only variants for 10,000 steps each, while the sequential variant receives 20,000 steps. The result therefore does not isolate the proposed curriculum from additional training. Both single-stage controls should receive the same total budget.

FVDtarget is also unusually fragile here: it fits a distribution of 400-dimensional I3D features separately for each subject, with only 581 clips across 119 subjects, or about five clips per subject on average. With so few videos, the covariance estimate has very low rank and the distance can be strongly affected by sampling. The paper provides neither uncertainty estimates nor a sample-count sensitivity analysis. The pooled FVDtest set is helpful, but it answers a different question and does not validate this subject-level estimate. Multiple generation seeds and subject-level confidence intervals would make the reported differences more convincing. Finally, encoding eight references once in 7.9 seconds does not give the total inference cost; the three guidance predictions per denoising step also need to be included.

I see a useful conditioning method, but the evidence for persistent, person-specific behavior is not yet strong enough for acceptance.

Questions for authors
  1. Can you evaluate reference and target clips from different recording sessions of the same person? How much of the improvement survives changes in context, framing and clothing?
  2. With the same video encoder, what happens when reference frames are shuffled or a single frame is repeated? Can behavior be identified independently of facial appearance?
  3. Do the curriculum gains remain when all variants receive 20,000 optimization steps?
  4. What are the per-subject clip counts and confidence intervals for FVDtarget? Please report sensitivity to sample count, multiple generation seeds, and total generation time/memory for the complete guidance procedure.

Reference

[1] Generate Your Talking Avatar from Video Reference (TAVR). https://arxiv.org/html/2604.27918

1 comment

Thank you for recognizing the value of behavioral personalization, the improvements in expression and pose matching, the fixed-size conditioning design, and the detailed experimental documentation.

W1 / Q1: References across recordings and recording conditions

Reference clips are not restricted to segments of the target recording. They also come from different recordings of the same person, potentially wearing different clothes. However, lighting and scene remain very similar due to the nature of the dataset. We clarify it in Appendix B.

W1 / Q2: Temporal information and appearance-independent behavior

Following Reviewer GFRH's suggestion, we have added landmark-velocity Fréchet distance and a face-normalized variant that removes translation and scale changes. Our method achieves the lowest face-normalized velocity distance and an image-coordinate velocity distance close to the best result. Adding our conditioning to FramePack improves both velocity metrics while largely preserving appearance similarity.

With the same trained model and encoder, shuffling reference frames at inference leaves appearance similarity relatively unchanged but worsens behavioral matching below the image-only baseline. This supports reliance on temporal cues, although the distribution shift prevents interpreting it as proof of appearance-independent behavior. Our tokens jointly encode appearance and behavior. We will provide the cross-identity experiment requested by Reviewer Xpxa as an additional diagnostic.

The repeated-frame control and an appearance-independent behavioral-identification study are not included in the current results.

W2 / Q3: Training-budget controls

We selected the 10,000-step single-stage checkpoints because further training did not improve performance. Cross-clip-only training even began to degrade. These observations explain our checkpoint choice but do not constitute a completed equal-budget comparison at 20,000 steps.

W2 / Q4: Per-subject FVD and generation seeds

Thank you for highlighting the sample-size limitation of per-subject FVD. We have removed FVD_target and retain pooled FVD for dataset-level video assessment. We will also evaluate multiple generation seeds and report the variance, as suggested.

We hope the additional evidence and clarifications address your main concerns and would appreciate your reconsideration of the assessment.

Reviewer_XpxaOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes to personalize audio-driven talking head generation beyond appearance by modeling the characteristic behavior of a person. They use a V-JEPA encoder and a query-based identity encoder to compress several reference videos into a fixed number of identity tokens, and inject them into a frozen Wan-S2V model through extra cross-attention layers. The training has three stages, first adapting V-JEPA to talking head videos, then training with same-clip conditioning, and finally with cross-clip conditioning. Experiments on Seamless Interaction show better expression and head pose distribution than the image-only baseline.

Critical strengths
  1. The problem is well motivated and new. Most personalization works only focus on the appearance, and this paper points out that the behavior of a person, such as the head movement and expression habit, is also part of the identity. Figure 1 shows this clearly.
  2. The method they proposed is simple and practical. It does not need any per-subject fine-tuning, the generator is kept frozen, and the number of identity tokens is fixed no matter how many reference videos are given, so adding more references does not increase the cost of the generation.
Critical weaknesses
  1. The experiments cannot show the model really learns the behavior. All the reference videos, the appearance image and the audio are from the same person, and the reference clips are even from the same recording as the target. The improvement may just come from appearance. A cross-identity experiment is needed.
  2. Personally, I found that he method is basically a Q-Former on V-JEPA features plus IP-Adapter style cross-attention on Wan-S2V, and the dual CFG is from InstructPix2Pix. Also in Table 2 the proposed method alone is worse than the simple FramePack baseline on CSIM and FVD.
Questions for authors
  1. Can you show cross-identity results, where the reference videos are from person A and the appearance image is from person B? Which person does the generated motion follow?
  2. In the cross-clip training, the reference clips are from the same recording as the target. Can you test with references from a different session of the same person, with different clothes and lighting?
  3. I am still not very clear, why is two-stage training needed in Figure 3? Is there a simpler way, such as a larger identity guidance weight at the beginning?
  4. In Table 2, Ours alone is worse than Non-Direct Video Reference on CSIM and FVD. How do you explain this?
  5. All results use one fixed seed. Can you report the variance over several seeds, since the differences in Table 3 and Table 5 are very small.
  6. The demo looks nice and I will be very appreciated if author can address my concerns expecially the Q1.
2 comments

Thank you for recognizing the motivation and practical design, particularly the frozen generator, fixed-size conditioning, and absence of per-subject fine-tuning.

W1 / Q1: Behavioral conditioning and cross-identity references

Following Reviewer GFRH's suggestion, we have added landmark-velocity Fréchet distance and a face-normalized variant that removes translation and scale changes. Our method achieves the lowest face-normalized velocity distance and an image-coordinate velocity distance close to the best result. In the controlled FramePack comparison, adding our conditioning improves both velocity metrics while appearance similarity remains nearly unchanged. This supports a motion-matching benefit beyond the measured appearance similarity.

Our identity tokens jointly encode appearance and behavior. We will provide a cross-identity experiment to examine their respective influence.

Q2: References from different recording sessions

Reference clips are not restricted to segments of the target recording. They also come from different recordings of the same person, potentially wearing different clothes. However, lighting and scene remain very similar due to the nature of the dataset. We clarify it in Appendix B.

W2: Methodological contribution beyond existing components

Our contribution is a framework for conditioning a frozen video diffusion generator on behavioral observations from multiple unaligned clips. While the components build on prior work, effective conditioning requires investigating the reference representation, aggregation, training curriculum, and guidance. Independent self-supervised adaptation of V-JEPA is one example of these design choices. It substantially improves expression and head-pose matching, with the resulting representation analyzed in Section 4.5.

Q3: Why same-clip training before cross-clip training?

The pretrained generator already preserves appearance reasonably well, making the additional behavioral signal from unaligned references difficult to learn. Starting directly with cross-clip training led to weaker reliance on the new conditioning. Same-clip training first establishes that reliance through aligned supervision, which the cross-clip stage then builds on. Higher inference guidance amplifies learned conditioning but cannot replace this training process.

W2 / Q4: Appearance and FVD compared with FramePack

We agree that FramePack achieves better CSIM and FVD than Ours alone, while our method achieves better expression and pose matching. We recommend Non-Direct + Ours for practical generation because it retains FramePack's appearance advantage while improving behavioral matching and FVD, with a synchronization trade-off. The user study also favors this combination. We have made this recommendation explicit in Section 4.2.

Q5: Variation across generation seeds

We will evaluate multiple generation seeds and report the variance, as suggested.

Reviewer_XpxaOpenReview ↗

Lack of experiments evidence

Thanks for the response, and sorry for the late reply due to the cvpr deadline (you know :) ). The responses above have addressed most of my concerns, but I would still like to see the results of the cross-identity experiment in your revision. There are still two days left in the rebuttal period, so please try your best to add this experiment, as it would strongly enhance your claim. Finally, I will keep my current rating regardless of whether you can add this experiment, but please do try.

Reviewer_aLVNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a spatio-temporal identity conditioning approach for audio-driven talking-head generation that goes beyond appearance preservation to capture subject-specific behavioral traits. The method aggregates multiple reference videos of the target subject using a domain adapted VJEPA encoder and a Spatio Temporal Identity Encoder to produce a fixed set of identity tokens, which are injected via new cross-attention branches into a frozen Wan-S2V diffusion transformer. A three-stage training procedure and a dual classifier-free guidance schedule enable controllable identity influence. Experiments on a large-scale talking-head dataset show consistent improvements over appearance-only baselines and several reference-video systems, with ablations supporting the design choices.

Critical strengths
  • It introduces a principled spatio-temporal identity encoder that compresses multiple reference clips into fixed-size identity tokens, balancing efficiency and expressivity.

  • It proposes a well-motivated curriculum, from the same clip to cross clip, to bootstrap identity conditioning and then encourage behavior cues that recur across clips.

  • Ablations on the encoders, training schedule variants, guidance schemes, and number/content of references are comprehensive.

Critical weaknesses
  1. It lacks a direct, style-specific behavioral metric design to fully reflect your characteristic behavior claim, beyond per-clip fidelity.

  2. It is suggested to compare the proposed token compression method with reference-token conditioning in TAVR, which would strengthen the efficiency argument. Moreover, please provide a complexity analysis with TAVR.

  3. Related methods for identity preserving designs, like test-time conditioning adaptation methods, as well as existing identity-aware S2V/T2V works, are not deeply compared or discussed.

  4. I have some fairness concerns in system-level baselines. OmniTalker results are coming from a web interface, which may affect fairness. MimicTalk relies on per-subject optimization, and please discuss the details.

Questions for authors

Please refer to weaknesses.

2 comments

Thank you for recognizing the encoder's balance of efficiency and expressivity, the motivation for our training curriculum, and the comprehensive ablations.

W1: Direct evaluation of behavioral style

Our expression and pose metrics follow OmniTalker's speaking-style evaluation, comparing distributions rather than requiring reproduction of the exact target performance. They do not capture temporal order.

Following Reviewer GFRH's suggestion, we have added landmark-velocity Fréchet distance and a face-normalized variant that removes translation and scale changes. Our method achieves the lowest face-normalized velocity distance and an image-coordinate velocity distance close to the best result.

W2: Token compression and complexity relative to TAVR

Appendix G provides an analytical complexity comparison. A direct multi-reference extension of TAVR increases Reference Self-Attention FLOPs by 4.09%, 12.64%, and 31.28% for 2, 4, and 8 total reference windows relative to one. The four- and eight-window estimates extrapolate the measured token density. Our fixed set of identity tokens gives a 0% increase in per-step identity-conditioning FLOPs. Additional references increase only our one-time encoding and aggregation cost.

W3: Relationship to identity-preserving adaptation and S2V/T2V methods

We expanded Section 2 to compare conditioning mechanisms and adaptation requirements across image- and video-based identity-preserving methods, clarifying how our approach aggregates behavioral observations from multiple clips.

W4: Fairness and adaptation details of external baselines

Public code and pretrained checkpoints were unavailable to us, so we evaluated OmniTalker through its web interface. We distinguish these system-level results from the controlled Wan-S2V comparisons. For MimicTalk, we used the official implementation with its default per-subject adaptation parameters. We have added the optimization and inference settings to Appendix C, under "MimicTalk baseline."

We hope the additional evidence and clarifications address your main concerns and would appreciate your reconsideration of the assessment.

Reviewer_aLVNOpenReview ↗

Replying Official Comment by Authors

Thank the authors for the detailed rebuttal. My main concerns have been substantially addressed. Remaining reservations are that OmniTalker is still not a fully controlled baseline, MimicTalk uses a different adaptation regime, and distributional style metrics are imperfect. Given that the main weaknesses have been addressed with new evidence and clarifications, I am raising my rating from 2 to 3.

Reviewer_GFRHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a personalized talking-head generation framework that captures not only a subject’s visual appearance but also their characteristic facial expressions, head movements, and gestures. It uses a spatiotemporal identity encoder to aggregate multiple reference videos into fixed-size identity tokens, which condition a frozen audio-driven video diffusion model through additional cross-attention layers. Experiments demonstrate improvements in appearance similarity and subject-specific motion statistics over existing baselines, supported by ablation studies and human evaluations.

Critical strengths
  1. The empirical performance is good; there is consistent improvement over baselines;
  2. The authors did comprehensive ablation studies to investigate several important design choices, including the video encoder, training procedure, identity guidance, and number of reference videos.
Critical weaknesses
  1. The methodological novelty appears limited. The core objective of capturing personalized facial expressions and head-motion patterns has already been investigated in StyleTalk, StyleTalk++, and OmniTalker. The proposed architecture largely combines established techniques, including pretrained video encoders, Q-Former-style aggregation, and IP-Adapter-style cross-attention conditioning. In particular, extending existing speaking-style representations to multiple temporally unaligned reference clips through feature aggregation appears straightforward. Consequently, it is difficult to identify a substantial technical contribution beyond integrating established components into the Wan-S2V framework.
  2. The evaluation primarily relies on per-frame facial-expression, head-pose, and landmark distributions, which do not directly capture the temporal dynamics central to the paper's behavioral personalization claim. A straightforward improvement would be to compute the velocities of the extracted facial landmarks and evaluate their distributions using a Fréchet distance, analogous to the existing Mesh-FID metric. This would provide more direct evidence of whether the generated videos reproduce subject-specific movement characteristics, rather than merely matching static facial configurations.
  3. A closely related paper, Styletalk++, is not discussed or compared with.
Questions for authors

N/A

2 comments

Thank you for your constructive feedback and for recognizing the consistent empirical improvements over baselines and the comprehensive ablations of our design choices.

W1: Methodological novelty and distinction from prior work

Our contribution is a framework that learns behavioral conditioning from multiple unaligned reference clips, compresses them into fixed-size identity tokens, and conditions a frozen video diffusion generator without per-subject optimization. While individual components build on prior work, making them effective together requires investigating the reference representation, multi-clip aggregation, conditioning curriculum, and guidance, which we study through dedicated ablations. Encoder adaptation is one example: we fine-tune V-JEPA independently with its self-supervised objective, then freeze it during conditioning training, rather than fine-tuning it through the generator's loss. This substantially improves expression and head-pose matching, and Section 4.5 analyzes the resulting representation changes.

Unlike StyleTalk/StyleTalk++'s 3DMM-based generation, we condition a frozen video diffusion model on multiple unaligned clips. The OmniTalker web-interface outputs we evaluated closely retain reference head movements. Our conditioning supports new motion without prescribing a reference trajectory.

W2: Evaluation of temporal dynamics

Thank you for this suggestion. We have added landmark-velocity Fréchet distance, analogous to Mesh-FID, and a face-normalized variant that removes translation and scale changes. Our method achieves the lowest face-normalized velocity distance and an image-coordinate velocity distance close to the best result. Adding our conditioning to FramePack also improves both velocity metrics while largely preserving appearance similarity.

W3: Missing discussion and comparison with StyleTalk++

We have added StyleTalk++ to Section 2. No public code or pretrained weights were available to us for a direct comparison.

We hope the additional evidence and clarifications address your main concerns and would appreciate your reconsideration of the assessment.

Thank you for the detailed rebuttal and the updated manuscript.

I appreciate the addition of landmark-velocity Fréchet distances, which substantially addresses my concern about the lack of temporal-dynamics evaluation. The new results provide more direct evidence that the proposed method improves facial-motion dynamics. I also appreciate the expanded related-work discussion and the computational analysis of multi-reference conditioning.

While I still have some reservations regarding methodological novelty, I acknowledge that the paper presents a meaningful integration of established techniques into an effective framework for behavioral personalization. Although the central objective of capturing personalized facial expressions and head movements has been explored in prior work, the proposed combination of spatiotemporal video representations, multi-reference aggregation, and conditioning of a frozen video diffusion model offers practical value. Together with the additional experimental evidence provided in the rebuttal, I find the overall contribution sufficiently compelling to warrant a more positive assessment.

Therefore, I'd like to raise my rating to weak accept.

Comments on the paper

1 comment

We thank the reviewers for their insightful comments. We are encouraged by their recognition of the value of behavioral personalization, the practicality of our framework, and its consistent empirical improvements. We are particularly pleased that reviewers highlighted the principled identity encoder with its well-motivated training curriculum. They also appreciated the comprehensive ablations and detailed experimental documentation.

Following the reviewers' feedback, we added landmark-velocity Fréchet distance and a face-normalized variant that removes translation and scale changes. Our method achieves the lowest face-normalized velocity distance and an image-coordinate velocity distance close to the best result. Adding our conditioning to FramePack improves both velocity metrics while largely preserving appearance similarity.

For more specific questions or comments, we address them directly to the relevant reviewer.

We uploaded an updated version with highlighted changes. We believe the additional temporal evaluation and clarifications in the revised manuscript address the main concerns. We thank the reviewers for helping us strengthen the paper and hope they will reconsider their assessments in light of the new evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.